Jul 02 14:14:21.090180 ip-10-0-134-54 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 02 14:14:21.090208 ip-10-0-134-54 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 02 14:14:21.090218 ip-10-0-134-54 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 02 14:14:21.090548 ip-10-0-134-54 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 02 14:14:31.236846 ip-10-0-134-54 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 02 14:14:31.236864 ip-10-0-134-54 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 32738264640a476a897913f3b5dce7ce --
Jul 02 14:16:33.111752 ip-10-0-134-54 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 14:16:33.592173 ip-10-0-134-54 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 14:16:33.592173 ip-10-0-134-54 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 14:16:33.592173 ip-10-0-134-54 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 14:16:33.592173 ip-10-0-134-54 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 14:16:33.592173 ip-10-0-134-54 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 14:16:33.592843 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.592225    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 14:16:33.598746 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598729    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:16:33.598746 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598746    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598750    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598753    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598757    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598760    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598763    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598766    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598769    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598772    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598774    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598777    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598780    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598783    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598786    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598789    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598792    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598795    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598799    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598802    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598804    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:16:33.598814 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598807    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598810    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598813    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598815    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598818    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598821    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598824    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598827    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598830    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598833    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598836    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598838    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598841    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598845    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598850    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598853    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598858    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598862    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598866    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:16:33.599306 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598869    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598872    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598875    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598878    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598881    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598884    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598887    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598890    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598893    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598895    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598898    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598901    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598904    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598906    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598909    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598913    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598916    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598918    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598921    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598924    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:16:33.599842 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598927    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598929    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598932    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598935    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598938    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598940    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598943    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598946    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598949    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598952    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598954    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598957    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598960    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598962    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598965    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598968    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598971    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598973    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598976    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598979    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:16:33.600354 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598984    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598988    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598990    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598994    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.598997    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599000    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599002    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599007    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599010    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599013    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599015    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599018    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599021    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599024    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599027    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599029    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599032    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599035    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599038    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599786    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:16:33.601176 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599794    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599797    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599800    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599803    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599806    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599809    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599812    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599814    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599817    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599820    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599823    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599826    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599829    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599832    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599835    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599838    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599840    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599845    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599848    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599852    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:16:33.602004 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599855    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599858    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599861    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599864    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599867    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599869    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599872    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599875    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599878    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599880    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599884    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599886    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599889    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599892    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599895    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599898    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599901    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599903    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599906    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:16:33.602566 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599909    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599911    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599914    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599925    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599929    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599948    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599952    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599955    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599958    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599961    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599964    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599968    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599970    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599973    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599977    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599980    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599983    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599985    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599988    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599991    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:16:33.603045 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599993    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599996    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.599999    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600001    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600004    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600007    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600010    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600013    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600017    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600020    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600023    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600026    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600028    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600033    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600037    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600041    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600044    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600047    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600050    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:16:33.603589 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600055    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600058    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600061    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600063    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600067    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600072    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600075    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600078    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600081    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600084    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600086    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600089    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600092    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600095    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600097    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600100    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600103    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600105    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600108    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:16:33.604083 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.600111    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601115    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601128    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601134    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601140    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601145    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601151    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601159    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601164    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601186    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601192    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601196    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601201    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601206    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601211    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601215    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601220    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601225    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601231    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601236    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:16:33.604601 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601241    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601245    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601255    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601259    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601264    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601269    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601274    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601278    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601283    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601287    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601292    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601296    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601301    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601305    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601310    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601323    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601329    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601335    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601340    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601349    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:16:33.605143 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601354    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601358    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601363    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601368    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601373    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601377    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601388    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601392    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601397    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601401    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601406    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601412    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601416    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601421    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601425    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601430    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601434    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601439    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601443    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:16:33.605666 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601453    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601458    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601463    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601468    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601472    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601477    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601482    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601487    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601491    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601496    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601500    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601505    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601511    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601520    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601526    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601530    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601535    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601540    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601562    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601571    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:16:33.606148 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601577    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601584    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601591    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601597    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601608    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601615    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601620    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601626    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601631    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601636    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601641    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601645    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601650    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601655    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601660    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601664    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601674    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601679    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601684    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601691    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:16:33.606708 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.601696    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601848    2551 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601865    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601872    2551 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601878    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601918    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601924    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601932    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601939    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601943    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601946    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601950    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601965    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.601977    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602111    2551 flags.go:64] FLAG: --cgroup-root=""
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602118    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602121    2551 flags.go:64] FLAG: --client-ca-file=""
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602125    2551 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602128    2551 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602133    2551 flags.go:64] FLAG: --cluster-domain=""
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602136    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602140    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602143    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602147    2551 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 14:16:33.607227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602151    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602154    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602157    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602161    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602164    2551 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602167    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602170    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602174    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602178    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602182    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602185    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602189    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602192    2551 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602195    2551 flags.go:64] FLAG: --enable-server="true"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602198    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602204    2551 flags.go:64] FLAG: --event-burst="100"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602207    2551 flags.go:64] FLAG: --event-qps="50"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602210    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602213    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602217    2551 flags.go:64] FLAG: --eviction-hard=""
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602221    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602224    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602228    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602231    2551 flags.go:64] FLAG: --eviction-soft=""
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602235    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 14:16:33.607848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602238    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602241    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602244    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602247    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602250    2551 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602253    2551 flags.go:64] FLAG: --feature-gates=""
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602258    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602261    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602264    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602267    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602270    2551 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602274    2551 flags.go:64] FLAG: --help="false"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602277    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602280    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602283    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602286    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602290    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602294    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602300    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602303    2551 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602307    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602310    2551 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602313    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602317    2551 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 14:16:33.608497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602320    2551 flags.go:64] FLAG: --kube-reserved=""
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602323    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602325    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602329    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602332    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602335    2551 flags.go:64] FLAG: --lock-file=""
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602338    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602341    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602344    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602350    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602353    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602356    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602359    2551 flags.go:64] FLAG: --logging-format="text"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602362    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602366    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602369    2551 flags.go:64] FLAG: --manifest-url=""
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602372    2551 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602377    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602380    2551 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602385    2551 flags.go:64] FLAG: --max-pods="110"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602388    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602391    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602394    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602397    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602400    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 14:16:33.609175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602403    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602408    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602417    2551 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602420    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602423    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602426    2551 flags.go:64] FLAG: --pod-cidr=""
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602430    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602436    2551 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602439    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602442    2551 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602445    2551 flags.go:64] FLAG: --port="10250"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602448    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602451    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d8738f4760b58044"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602455    2551 flags.go:64] FLAG: --qos-reserved=""
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602458    2551 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602461    2551 flags.go:64] FLAG: --register-node="true"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602464    2551 flags.go:64] FLAG: --register-with-taints=""
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602468    2551 flags.go:64] FLAG: --registry-burst="10"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602471    2551 flags.go:64] FLAG: --registry-qps="5"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602474    2551 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602477    2551 flags.go:64] FLAG: --reserved-memory=""
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602482    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602485    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602488    2551 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602491    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 14:16:33.609903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602494    2551 flags.go:64] FLAG: --runonce="false"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602497    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602500    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602504    2551 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602507    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602510    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602513    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602516    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602522    2551 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602525    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602528    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602531    2551 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602534    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602537    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602540    2551 flags.go:64] FLAG: --system-cgroups=""
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602559    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602567    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602570    2551 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602573    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602578    2551 flags.go:64] FLAG: --tls-min-version=""
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602581    2551 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602584    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602587    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602590    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602593    2551 flags.go:64] FLAG: --v="2"
Jul 02 14:16:33.610609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602597    2551 flags.go:64] FLAG: --version="false"
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602602    2551 flags.go:64] FLAG: --vmodule=""
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602606    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.602610    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602737    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602742    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602746    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602749    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602751    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602754    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602757    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602760    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602763    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602765    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602768    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602773    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602775    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602779    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602783    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602787    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:16:33.611245 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602790    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602793    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602796    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602799    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602804    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602807    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602810    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602814    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602817    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602820    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602823    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602826    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602830    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602833    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602835    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602839    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602844    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602847    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602850    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602853    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:16:33.611785 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602856    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602859    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602862    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602865    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602869    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602872    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602875    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602879    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602882    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602885    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602888    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602890    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602893    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602897    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602899    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602902    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602905    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602907    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602910    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602913    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:16:33.612294 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602916    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602918    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602921    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602924    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602927    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602930    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602932    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602935    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602940    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602943    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602946    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602949    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602952    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602955    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602958    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602961    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602963    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602966    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602968    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:16:33.612819 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602972    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602976    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602979    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602982    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602984    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602987    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602990    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602993    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602995    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.602999    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603003    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603006    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603009    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603012    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603014    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603017    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603020    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603023    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603025    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603028    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:16:33.613325 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603030    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603034    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603038    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.603041    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.603631    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.610789    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.610805    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.610825    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.610831    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610886    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610892    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610896    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610899    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610903    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610905    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:16:33.613977 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610908    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610912    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610915    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610918    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610921    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610924    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610927    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610929    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610932    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610935    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610938    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610941    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610943    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610946    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610949    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610952    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610955    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610957    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610960    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610963    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:16:33.614378 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610966    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610969    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610972    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610975    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610979    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610981    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610984    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610987    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610990    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610993    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610996    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.610999    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611001    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611004    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611007    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611009    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611012    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611015    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611017    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611020    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:16:33.614911 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611023    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611026    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611029    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611031    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611036    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611040    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611043    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611047    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611050    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611053    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611056    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611059    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611063    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611066    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611070    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611076    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611079    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611082    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611085    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:16:33.615468 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611088    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611091    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611094    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611096    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611099    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611102    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611105    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611108    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611110    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611113    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611116    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611119    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611121    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611124    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611127    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611130    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611132    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611135    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611138    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:16:33.615985 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611141    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611144    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611146    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611149    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611151    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611154    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611157    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611160    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611163    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611167    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611171    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611174    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611177    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611180    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611183    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:16:33.616466 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.611188    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611304    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611308    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611311    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611314    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611319    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611323    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611326    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611329    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611331    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611334    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611337    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611340    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611342    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611345    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611348    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611351    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611353    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611356    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:16:33.616899 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611359    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611362    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611365    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611367    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611370    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611373    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611376    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611379    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611382    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611385    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611388    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611390    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611393    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611397    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611401    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611404    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611407    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611410    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611412    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:16:33.617399 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611415    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611418    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611420    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611423    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611426    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611428    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611431    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611433    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611436    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611439    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611441    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611444    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611447    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611449    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611452    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611455    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611458    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611460    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611463    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611465    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:16:33.617900 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611474    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611477    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611479    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611482    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611485    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611488    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611491    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611493    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611496    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611498    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611501    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611504    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611507    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611509    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611512    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611515    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611517    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611520    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611523    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611525    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:16:33.618404 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611528    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611530    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611533    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611536    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611540    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611557    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611561    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611563    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611566    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611569    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611571    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611574    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611577    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611580    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611583    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611585    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611588    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611590    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611593    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611596    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:16:33.618935 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611599    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:16:33.619445 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:33.611601    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:16:33.619445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.611606    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 14:16:33.619445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.611712    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 14:16:33.619445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.614889    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 14:16:33.619445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.616454    2551 server.go:1019] "Starting client certificate rotation"
Jul 02 14:16:33.619445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.616571    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 14:16:33.619445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.616608    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 14:16:33.619798 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.619784    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 14:16:33.623746 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.623732    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 14:16:33.635690 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.635670    2551 log.go:25] "Validated CRI v1 runtime API"
Jul 02 14:16:33.638409 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.638394    2551 log.go:25] "Validated CRI v1 image API"
Jul 02 14:16:33.639192 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.639178    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 14:16:33.641697 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.641671    2551 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8f8a8c18-2332-443f-b1cb-2c6f79853393:/dev/nvme0n1p3 94cf6c0c-7c51-46ff-930d-cde7c0f987fa:/dev/nvme0n1p4]
Jul 02 14:16:33.641762 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.641697    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 14:16:33.648422 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.648302    2551 manager.go:216] Machine: {Timestamp:2026-07-02 14:16:33.646964621 +0000 UTC m=+0.410349936 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096747 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23818251ef27f1d47ba3302991f469 SystemUUID:ec238182-51ef-27f1-d47b-a3302991f469 BootID:32738264-640a-476a-8979-13f3b5dce7ce Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:73:bb:01:42:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:73:bb:01:42:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:e5:84:20:f2:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 14:16:33.648422 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.648404    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 14:16:33.648580 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.648482    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 14:16:33.652485 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.652462    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 14:16:33.652634 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.652487    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-54.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 14:16:33.653887 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.653874    2551 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 14:16:33.653948 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.653890    2551 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 14:16:33.653948 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.653904    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 14:16:33.654642 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.654631    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 14:16:33.655853 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.655842    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 14:16:33.656359 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.656341    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 14:16:33.659926 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.659914    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 14:16:33.660051 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.660041    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 14:16:33.660084 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.660073    2551 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 14:16:33.660113 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.660087    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 14:16:33.660113 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.660099    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 14:16:33.660113 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.660110    2551 kubelet.go:398] "Adding apiserver pod source"
Jul 02 14:16:33.660207 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.660118    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 14:16:33.660467 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.660454    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 14:16:33.660522 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.660478    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 14:16:33.662757 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.662739    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 14:16:33.664141 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664126    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 14:16:33.664200 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664158    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 14:16:33.664819 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664808    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 14:16:33.664857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664824    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 14:16:33.664857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664831    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 14:16:33.664857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664837    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 14:16:33.664857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664847    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 14:16:33.664857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664853    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 14:16:33.665011 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664910    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 14:16:33.665011 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664923    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 14:16:33.665011 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664932    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 14:16:33.665011 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664938    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 14:16:33.665011 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664955    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 14:16:33.665011 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.664963    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 14:16:33.666048 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.666038    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 14:16:33.666048 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.666049    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 14:16:33.669352 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.669337    2551 server.go:1268] "Started kubelet"
Jul 02 14:16:33.669447 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.669413    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 14:16:33.669886 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.669603    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 14:16:33.669886 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.669667    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 14:16:33.670226 ip-10-0-134-54 systemd[1]: Started Kubernetes Kubelet.
Jul 02 14:16:33.671328 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.671284    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 14:16:33.672294 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.672258    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-54.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 14:16:33.672503 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.672446    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-54.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 14:16:33.672633 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.672610    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 14:16:33.672824 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.672809    2551 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 14:16:33.675880 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.675861    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 14:16:33.678412 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.678393    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 14:16:33.678412 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.678406    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 14:16:33.678679 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.677671    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-54.ec2.internal.18be7eaf418b1eee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-54.ec2.internal,UID:ip-10-0-134-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-54.ec2.internal,},FirstTimestamp:2026-07-02 14:16:33.669308142 +0000 UTC m=+0.432693459,LastTimestamp:2026-07-02 14:16:33.669308142 +0000 UTC m=+0.432693459,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-54.ec2.internal,}"
Jul 02 14:16:33.678806 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.678683    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:33.679304 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.679274    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 14:16:33.679304 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.679276    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 14:16:33.679445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.679318    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 14:16:33.679445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.679391    2551 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 14:16:33.679445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.679401    2551 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 14:16:33.679611 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.679453    2551 factory.go:55] Registering systemd factory
Jul 02 14:16:33.679611 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.679497    2551 factory.go:223] Registration of the systemd container factory successfully
Jul 02 14:16:33.680796 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.680771    2551 factory.go:153] Registering CRI-O factory
Jul 02 14:16:33.680876 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.680799    2551 factory.go:223] Registration of the crio container factory successfully
Jul 02 14:16:33.680876 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.680862    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 14:16:33.682608 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.682585    2551 factory.go:103] Registering Raw factory
Jul 02 14:16:33.682736 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.682722    2551 manager.go:1196] Started watching for new ooms in manager
Jul 02 14:16:33.683670 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.683646    2551 manager.go:349] Starting recovery of all containers
Jul 02 14:16:33.684582 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.684554    2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-54.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 14:16:33.684726 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.684697    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 14:16:33.695417 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.695401    2551 manager.go:354] Recovery completed
Jul 02 14:16:33.699995 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.699895    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:16:33.701744 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.701722    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:16:33.701826 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.701755    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:16:33.701826 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.701766    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:16:33.702316 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.702302    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 14:16:33.702316 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.702314    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 14:16:33.702412 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.702330    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 14:16:33.702412 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.702349    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:16:33.703074 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.703057    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:16:33.703128 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.703085    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:16:33.703128 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.703101    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:16:33.704348 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.704280    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-54.ec2.internal.18be7eaf437a063d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-54.ec2.internal,UID:ip-10-0-134-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-54.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-54.ec2.internal,},FirstTimestamp:2026-07-02 14:16:33.701742141 +0000 UTC m=+0.465127459,LastTimestamp:2026-07-02 14:16:33.701742141 +0000 UTC m=+0.465127459,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-54.ec2.internal,}"
Jul 02 14:16:33.704615 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.704601    2551 policy_none.go:49] "None policy: Start"
Jul 02 14:16:33.704668 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.704627    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 14:16:33.704668 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.704641    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 14:16:33.706701 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.706689    2551 policy_none.go:47] "Start"
Jul 02 14:16:33.716975 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.716893    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-54.ec2.internal.18be7eaf437a4e2b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-54.ec2.internal,UID:ip-10-0-134-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-54.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-54.ec2.internal,},FirstTimestamp:2026-07-02 14:16:33.701760555 +0000 UTC m=+0.465145870,LastTimestamp:2026-07-02 14:16:33.701760555 +0000 UTC m=+0.465145870,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-54.ec2.internal,}"
Jul 02 14:16:33.724259 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.724171    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-54.ec2.internal.18be7eaf437a735d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-54.ec2.internal,UID:ip-10-0-134-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-54.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-54.ec2.internal,},FirstTimestamp:2026-07-02 14:16:33.701770077 +0000 UTC m=+0.465155392,LastTimestamp:2026-07-02 14:16:33.701770077 +0000 UTC m=+0.465155392,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-54.ec2.internal,}"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.734403    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-54.ec2.internal.18be7eaf437a063d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-54.ec2.internal.18be7eaf437a063d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-54.ec2.internal,UID:ip-10-0-134-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-54.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-54.ec2.internal,},FirstTimestamp:2026-07-02 14:16:33.701742141 +0000 UTC m=+0.465127459,LastTimestamp:2026-07-02 14:16:33.703072521 +0000 UTC m=+0.466457835,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-54.ec2.internal,}"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.741813    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.743040    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.743082    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.743101    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.743148    2551 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.743764    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-54.ec2.internal.18be7eaf437a4e2b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-54.ec2.internal.18be7eaf437a4e2b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-54.ec2.internal,UID:ip-10-0-134-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-54.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-54.ec2.internal,},FirstTimestamp:2026-07-02 14:16:33.701760555 +0000 UTC m=+0.465145870,LastTimestamp:2026-07-02 14:16:33.703091944 +0000 UTC m=+0.466477261,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-54.ec2.internal,}"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.745226    2551 manager.go:337] "Starting Device Plugin manager"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.745257    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.745269    2551 server.go:87] "Starting device plugin registration server"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.745521    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.745534    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.745755    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.745869    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.745882    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.746381    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.746423    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.753190    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f5kv6"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.754433    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Jul 02 14:16:33.760010 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.754541    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-54.ec2.internal.18be7eaf437a735d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-54.ec2.internal.18be7eaf437a735d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-54.ec2.internal,UID:ip-10-0-134-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-54.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-54.ec2.internal,},FirstTimestamp:2026-07-02 14:16:33.701770077 +0000 UTC m=+0.465155392,LastTimestamp:2026-07-02 14:16:33.703106309 +0000 UTC m=+0.466491623,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-54.ec2.internal,}"
Jul 02 14:16:33.760694 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.757964    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-54.ec2.internal.18be7eaf4637ed36  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-54.ec2.internal,UID:ip-10-0-134-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-134-54.ec2.internal,},FirstTimestamp:2026-07-02 14:16:33.747742006 +0000 UTC m=+0.511127307,LastTimestamp:2026-07-02 14:16:33.747742006 +0000 UTC m=+0.511127307,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-54.ec2.internal,}"
Jul 02 14:16:33.766821 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.765975    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f5kv6"
Jul 02 14:16:33.847666 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.847600    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:16:33.848390 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.847617    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal"]
Jul 02 14:16:33.848464 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.848447    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:16:33.848527 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.848517    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:16:33.848609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.848546    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:16:33.848609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.848557    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:16:33.848609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.848580    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.849526 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.849513    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:16:33.849594 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.849539    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:16:33.849594 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.849550    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:16:33.851191 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.851130    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-54.ec2.internal.18be7eaf437a063d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-54.ec2.internal.18be7eaf437a063d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-54.ec2.internal,UID:ip-10-0-134-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-54.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-54.ec2.internal,},FirstTimestamp:2026-07-02 14:16:33.701742141 +0000 UTC m=+0.465127459,LastTimestamp:2026-07-02 14:16:33.848530551 +0000 UTC m=+0.611915867,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-54.ec2.internal,}"
Jul 02 14:16:33.851527 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.851513    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:16:33.851634 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.851618    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.851674 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.851653    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:16:33.852142 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.852125    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:16:33.852142 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.852139    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:16:33.852237 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.852157    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:16:33.852237 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.852156    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:16:33.852237 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.852183    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:16:33.852237 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.852167    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:16:33.854129 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.854080    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.854129 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.854110    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:16:33.858400 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.858379    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:16:33.858519 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.858408    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:16:33.858519 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.858424    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:16:33.860818 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.860800    2551 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-134-54.ec2.internal\" not found" node="ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.872987 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.872969    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-54.ec2.internal\" not found" node="ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.878845 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.878829    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-54.ec2.internal\" not found" node="ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.880999 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.880983    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b9e4c4239276a54ecd642935c5ffe6cd-config\") pod \"kube-apiserver-proxy-ip-10-0-134-54.ec2.internal\" (UID: \"b9e4c4239276a54ecd642935c5ffe6cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.881056 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.881007    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e2f6dd71485cba712a5a6213acf7cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal\" (UID: \"5e2f6dd71485cba712a5a6213acf7cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.881056 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.881026    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e2f6dd71485cba712a5a6213acf7cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal\" (UID: \"5e2f6dd71485cba712a5a6213acf7cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.892234 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:33.892215    2551 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-54.ec2.internal\" not found" node="ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.981309 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.981283    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e2f6dd71485cba712a5a6213acf7cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal\" (UID: \"5e2f6dd71485cba712a5a6213acf7cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.981424 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.981316    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e2f6dd71485cba712a5a6213acf7cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal\" (UID: \"5e2f6dd71485cba712a5a6213acf7cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.981424 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.981334    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b9e4c4239276a54ecd642935c5ffe6cd-config\") pod \"kube-apiserver-proxy-ip-10-0-134-54.ec2.internal\" (UID: \"b9e4c4239276a54ecd642935c5ffe6cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.981424 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.981398    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b9e4c4239276a54ecd642935c5ffe6cd-config\") pod \"kube-apiserver-proxy-ip-10-0-134-54.ec2.internal\" (UID: \"b9e4c4239276a54ecd642935c5ffe6cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.981424 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.981412    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e2f6dd71485cba712a5a6213acf7cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal\" (UID: \"5e2f6dd71485cba712a5a6213acf7cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:33.981579 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:33.981398    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5e2f6dd71485cba712a5a6213acf7cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal\" (UID: \"5e2f6dd71485cba712a5a6213acf7cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:34.064431 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.064405    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:16:34.065348 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.065331    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:16:34.065407 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.065361    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:16:34.065407 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.065371    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:16:34.065407 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.065396    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-54.ec2.internal"
Jul 02 14:16:34.074037 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.074020    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-54.ec2.internal"
Jul 02 14:16:34.074110 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:34.074042    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-54.ec2.internal\": node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:34.090231 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:34.090204    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:34.177071 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.177013    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:34.183048 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.183032    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:34.190373 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:34.190356    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:34.291309 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:34.291273    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:34.392338 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:34.392311    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:34.493527 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:34.493429    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:34.594020 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:34.593990    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:34.616415 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.616375    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 14:16:34.694849 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:34.694828    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-54.ec2.internal\" not found"
Jul 02 14:16:34.760237 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.760216    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:16:34.761057 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.761042    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:16:34.761122 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.761074    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:16:34.761122 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.761089    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:16:34.771946 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.771905    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 14:11:34 +0000 UTC" deadline="2027-12-24 04:41:37.401675168 +0000 UTC"
Jul 02 14:16:34.772023 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.771959    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12950h25m2.629737974s"
Jul 02 14:16:34.783038 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.783017    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:16:34.800602 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.800582    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:16:34.822723 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.822705    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:16:34.882647 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.882609    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:34.896175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.896157    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 14:16:34.897144 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.897130    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal"
Jul 02 14:16:34.911276 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:34.911258    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 14:16:35.125630 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.125567    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:16:35.544318 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:35.544290    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e4c4239276a54ecd642935c5ffe6cd.slice/crio-29277627f0b91b9a36bad4979b30e1c164fb151b4bafb916d85bb0243d33cc19 WatchSource:0}: Error finding container 29277627f0b91b9a36bad4979b30e1c164fb151b4bafb916d85bb0243d33cc19: Status 404 returned error can't find the container with id 29277627f0b91b9a36bad4979b30e1c164fb151b4bafb916d85bb0243d33cc19
Jul 02 14:16:35.548933 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.548918    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:16:35.663456 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.663424    2551 apiserver.go:52] "Watching apiserver"
Jul 02 14:16:35.671351 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.671333    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 14:16:35.671744 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.671725    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-6krdm","kube-system/global-pull-secret-syncer-qzshm","kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j","openshift-cluster-node-tuning-operator/tuned-rcn6r","openshift-dns/node-resolver-z2vnx","openshift-image-registry/node-ca-4f9jp","openshift-multus/multus-n6zrx","openshift-multus/network-metrics-daemon-gr9j2","kube-system/konnectivity-agent-wzk2v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal","openshift-multus/multus-additional-cni-plugins-jn8cz","openshift-network-diagnostics/network-check-target-jpqgc","openshift-network-operator/iptables-alerter-8qtb7"]
Jul 02 14:16:35.675512 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.675496    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.677721 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.677698    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:35.677821 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.677790    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:35.679846 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.679792    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 14:16:35.679944 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.679932    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.683285 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.682600    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 14:16:35.683285 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.682684    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.683285 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.682850    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 14:16:35.683285 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.683218    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 14:16:35.683785 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.683576    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vtdsj\""
Jul 02 14:16:35.683785 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.683583    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 14:16:35.683785 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.683667    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 14:16:35.685054 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.685038    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.687522 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.687506    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.689924 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.689908    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:35.690015 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.689988    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:35.692133 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.692116    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 14:16:35.692517 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.692503    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.695126 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695108    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:35.695197 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695138    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf27d440-b641-4efc-9a92-273c9bbef679-serviceca\") pod \"node-ca-4f9jp\" (UID: \"bf27d440-b641-4efc-9a92-273c9bbef679\") " pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.695197 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695163    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d29ddb9d-0592-481b-96f9-90acc879de60-iptables-alerter-script\") pod \"iptables-alerter-8qtb7\" (UID: \"d29ddb9d-0592-481b-96f9-90acc879de60\") " pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.695197 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.695165    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:35.695197 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695179    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d29ddb9d-0592-481b-96f9-90acc879de60-host-slash\") pod \"iptables-alerter-8qtb7\" (UID: \"d29ddb9d-0592-481b-96f9-90acc879de60\") " pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.695402 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695198    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf27d440-b641-4efc-9a92-273c9bbef679-host\") pod \"node-ca-4f9jp\" (UID: \"bf27d440-b641-4efc-9a92-273c9bbef679\") " pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.695402 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695222    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:35.695402 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695273    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l9w6\" (UniqueName: \"kubernetes.io/projected/d29ddb9d-0592-481b-96f9-90acc879de60-kube-api-access-7l9w6\") pod \"iptables-alerter-8qtb7\" (UID: \"d29ddb9d-0592-481b-96f9-90acc879de60\") " pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.695402 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695307    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.695402 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695338    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-socket-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.695402 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695363    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-registration-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.695651 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695406    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-sys-fs\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.695651 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695431    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drhc5\" (UniqueName: \"kubernetes.io/projected/bf27d440-b641-4efc-9a92-273c9bbef679-kube-api-access-drhc5\") pod \"node-ca-4f9jp\" (UID: \"bf27d440-b641-4efc-9a92-273c9bbef679\") " pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.695651 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695446    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.695651 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695464    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drwb\" (UniqueName: \"kubernetes.io/projected/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-kube-api-access-2drwb\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.695651 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.695521    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-device-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.697561 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.697540    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:35.700234 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.700219    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.702201 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.702171    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kcc7b\""
Jul 02 14:16:35.702301 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.702231    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 14:16:35.702301 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.702243    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 14:16:35.702301 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.702282    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 14:16:35.702457 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.702443    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 14:16:35.702541 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.702484    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 14:16:35.702621 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.702601    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 14:16:35.702903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.702888    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.703522 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703503    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 14:16:35.703522 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703516    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 14:16:35.703684 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703507    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 14:16:35.703684 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703503    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 14:16:35.703684 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703623    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 14:16:35.703838 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703762    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s6qtk\""
Jul 02 14:16:35.703838 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703777    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6j9c9\""
Jul 02 14:16:35.703838 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703798    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 14:16:35.703838 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703827    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n2xxv\""
Jul 02 14:16:35.704134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703866    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-66drc\""
Jul 02 14:16:35.704134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.703938    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 14:16:35.704238 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.704194    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 14:16:35.704238 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.704226    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n4cff\""
Jul 02 14:16:35.708728 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.708706    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 14:16:35.708728 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.708723    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 14:16:35.708894 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.708808    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 14:16:35.709158 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.709142    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-58zgp\""
Jul 02 14:16:35.713668 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.713651    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 14:16:35.713946 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.713932    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 14:16:35.718589 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.718564    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 14:16:35.718702 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.718577    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 14:16:35.718702 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.718649    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bm92k\""
Jul 02 14:16:35.718702 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.718691    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 14:16:35.749280 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.749231    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal" event={"ID":"b9e4c4239276a54ecd642935c5ffe6cd","Type":"ContainerStarted","Data":"29277627f0b91b9a36bad4979b30e1c164fb151b4bafb916d85bb0243d33cc19"}
Jul 02 14:16:35.781487 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.781463    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 14:16:35.796606 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796589    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-modprobe-d\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.796681 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796615    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-cni-dir\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.796681 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796632    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-system-cni-dir\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.796681 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796650    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fae8e80b-baa0-4195-9657-aefc73c89147-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.796681 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796667    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-systemd-units\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.796829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796697    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.796829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796735    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-system-cni-dir\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.796829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796756    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-var-lib-cni-multus\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.796829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-daemon-config\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.796829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796788    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdvsn\" (UniqueName: \"kubernetes.io/projected/bde91108-8391-455f-ae1f-ed867f2e9bdf-kube-api-access-fdvsn\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.796829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796804    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26m22\" (UniqueName: \"kubernetes.io/projected/fae8e80b-baa0-4195-9657-aefc73c89147-kube-api-access-26m22\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.796829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796819    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-cni-netd\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797064 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796833    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dfb74e7-e835-4ed2-ab1d-55985f662845-ovnkube-script-lib\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797064 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796848    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-run-multus-certs\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.797064 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796887    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-run\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.797064 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796930    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d29ddb9d-0592-481b-96f9-90acc879de60-iptables-alerter-script\") pod \"iptables-alerter-8qtb7\" (UID: \"d29ddb9d-0592-481b-96f9-90acc879de60\") " pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.797064 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.796967    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-kubernetes\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.797064 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797017    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a885d1ba-0164-40a1-a476-b982c984dbe7-agent-certs\") pod \"konnectivity-agent-wzk2v\" (UID: \"a885d1ba-0164-40a1-a476-b982c984dbe7\") " pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:35.797064 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797046    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgm7t\" (UniqueName: \"kubernetes.io/projected/79b5c08b-176c-4374-a72b-97d488dcc046-kube-api-access-bgm7t\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:35.797064 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797066    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-run-netns\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797082    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-cnibin\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.797330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797110    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-sys-fs\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.797330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797139    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-var-lib-cni-bin\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.797330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797173    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-sys-fs\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.797330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797199    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-var-lib-openvswitch\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797238    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-etc-openvswitch\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797275    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dfb74e7-e835-4ed2-ab1d-55985f662845-env-overrides\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797301    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dfb74e7-e835-4ed2-ab1d-55985f662845-ovn-node-metrics-cert\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797336    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjt9r\" (UniqueName: \"kubernetes.io/projected/2dfb74e7-e835-4ed2-ab1d-55985f662845-kube-api-access-sjt9r\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797362    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpn9k\" (UniqueName: \"kubernetes.io/projected/9c7b1df8-a5d3-4087-9362-725f43f825d6-kube-api-access-zpn9k\") pod \"node-resolver-z2vnx\" (UID: \"9c7b1df8-a5d3-4087-9362-725f43f825d6\") " pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797387    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-log-socket\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797409    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-run-ovn-kubernetes\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797432    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3e26069-93f9-43ce-9605-f1da44d37f09-dbus\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797451    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797453    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d29ddb9d-0592-481b-96f9-90acc879de60-iptables-alerter-script\") pod \"iptables-alerter-8qtb7\" (UID: \"d29ddb9d-0592-481b-96f9-90acc879de60\") " pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797466    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-os-release\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797497    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fae8e80b-baa0-4195-9657-aefc73c89147-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797524    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-sys\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797550    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-sysctl-conf\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797566    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-os-release\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797580    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zztp4\" (UniqueName: \"kubernetes.io/projected/daa7d524-66f0-4937-83f2-8a7a0766dd9a-kube-api-access-zztp4\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.797609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797600    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a885d1ba-0164-40a1-a476-b982c984dbe7-konnectivity-ca\") pod \"konnectivity-agent-wzk2v\" (UID: \"a885d1ba-0164-40a1-a476-b982c984dbe7\") " pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797629    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-cnibin\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797653    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-run-k8s-cni-cncf-io\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797670    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-etc-kubernetes\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797691    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-run-systemd\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797711    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-node-log\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797739    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drhc5\" (UniqueName: \"kubernetes.io/projected/bf27d440-b641-4efc-9a92-273c9bbef679-kube-api-access-drhc5\") pod \"node-ca-4f9jp\" (UID: \"bf27d440-b641-4efc-9a92-273c9bbef679\") " pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797780    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797853    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-socket-dir-parent\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797876    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-hostroot\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797891    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797894    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3e26069-93f9-43ce-9605-f1da44d37f09-kubelet-config\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797933    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-lib-modules\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797955    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-host\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797972    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c7b1df8-a5d3-4087-9362-725f43f825d6-tmp-dir\") pod \"node-resolver-z2vnx\" (UID: \"9c7b1df8-a5d3-4087-9362-725f43f825d6\") " pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.797994    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-slash\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798016    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-var-lib-kubelet\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.798134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798036    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf27d440-b641-4efc-9a92-273c9bbef679-serviceca\") pod \"node-ca-4f9jp\" (UID: \"bf27d440-b641-4efc-9a92-273c9bbef679\") " pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798050    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/daa7d524-66f0-4937-83f2-8a7a0766dd9a-tmp\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798066    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf27d440-b641-4efc-9a92-273c9bbef679-host\") pod \"node-ca-4f9jp\" (UID: \"bf27d440-b641-4efc-9a92-273c9bbef679\") " pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798092    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798114    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l9w6\" (UniqueName: \"kubernetes.io/projected/d29ddb9d-0592-481b-96f9-90acc879de60-kube-api-access-7l9w6\") pod \"iptables-alerter-8qtb7\" (UID: \"d29ddb9d-0592-481b-96f9-90acc879de60\") " pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798128    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf27d440-b641-4efc-9a92-273c9bbef679-host\") pod \"node-ca-4f9jp\" (UID: \"bf27d440-b641-4efc-9a92-273c9bbef679\") " pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798147    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798206    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798263    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-registration-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798292    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bde91108-8391-455f-ae1f-ed867f2e9bdf-cni-binary-copy\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798316    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-registration-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798321    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drwb\" (UniqueName: \"kubernetes.io/projected/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-kube-api-access-2drwb\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798350    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-tuned\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798370    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c7b1df8-a5d3-4087-9362-725f43f825d6-hosts-file\") pod \"node-resolver-z2vnx\" (UID: \"9c7b1df8-a5d3-4087-9362-725f43f825d6\") " pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798373    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf27d440-b641-4efc-9a92-273c9bbef679-serviceca\") pod \"node-ca-4f9jp\" (UID: \"bf27d440-b641-4efc-9a92-273c9bbef679\") " pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798386    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-run-openvswitch\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798424    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-device-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.798760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798455    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-sysctl-d\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798513    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-run-netns\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798528    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-device-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798533    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-conf-dir\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798563    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-kubelet\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798579    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-run-ovn\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798602    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dfb74e7-e835-4ed2-ab1d-55985f662845-ovnkube-config\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798626    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d29ddb9d-0592-481b-96f9-90acc879de60-host-slash\") pod \"iptables-alerter-8qtb7\" (UID: \"d29ddb9d-0592-481b-96f9-90acc879de60\") " pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798652    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-sysconfig\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798683    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d29ddb9d-0592-481b-96f9-90acc879de60-host-slash\") pod \"iptables-alerter-8qtb7\" (UID: \"d29ddb9d-0592-481b-96f9-90acc879de60\") " pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798696    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-systemd\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798720    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-var-lib-kubelet\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798738    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fae8e80b-baa0-4195-9657-aefc73c89147-cni-binary-copy\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798755    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798777    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798797    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-cni-bin\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798813    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-socket-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.799290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.798927    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-socket-dir\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.809464 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.809445    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vhbxd"
Jul 02 14:16:35.811483 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.811455    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 14:16:35.813376 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.813356    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:16:35.813376 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.813376    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:16:35.813561 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.813385    2551 projected.go:196] Error preparing data for projected volume kube-api-access-65r9n for pod openshift-network-diagnostics/network-check-target-jpqgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:35.813561 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.813438    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n podName:065c8aea-2847-43cd-bff7-ba1e706a032d nodeName:}" failed. No retries permitted until 2026-07-02 14:16:36.313419994 +0000 UTC m=+3.076805300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-65r9n" (UniqueName: "kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n") pod "network-check-target-jpqgc" (UID: "065c8aea-2847-43cd-bff7-ba1e706a032d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:35.815030 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.815012    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drwb\" (UniqueName: \"kubernetes.io/projected/0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3-kube-api-access-2drwb\") pod \"aws-ebs-csi-driver-node-rcw4j\" (UID: \"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:35.815133 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.815114    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drhc5\" (UniqueName: \"kubernetes.io/projected/bf27d440-b641-4efc-9a92-273c9bbef679-kube-api-access-drhc5\") pod \"node-ca-4f9jp\" (UID: \"bf27d440-b641-4efc-9a92-273c9bbef679\") " pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.815590 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.815574    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l9w6\" (UniqueName: \"kubernetes.io/projected/d29ddb9d-0592-481b-96f9-90acc879de60-kube-api-access-7l9w6\") pod \"iptables-alerter-8qtb7\" (UID: \"d29ddb9d-0592-481b-96f9-90acc879de60\") " pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.818199 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.818172    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vhbxd"
Jul 02 14:16:35.838080 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:35.838056    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2f6dd71485cba712a5a6213acf7cad.slice/crio-c18871a162d1fcbc2ef09315dd9ea8ef236abf1f48e5a6d442907bb1bb659dc3 WatchSource:0}: Error finding container c18871a162d1fcbc2ef09315dd9ea8ef236abf1f48e5a6d442907bb1bb659dc3: Status 404 returned error can't find the container with id c18871a162d1fcbc2ef09315dd9ea8ef236abf1f48e5a6d442907bb1bb659dc3
Jul 02 14:16:35.900107 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900078    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-sysctl-d\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.900107 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900105    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-run-netns\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900121    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-conf-dir\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900138    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-kubelet\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900161    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-run-ovn\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900176    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dfb74e7-e835-4ed2-ab1d-55985f662845-ovnkube-config\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900184    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-run-netns\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900192    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-sysconfig\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900206    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-conf-dir\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900212    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-kubelet\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900232    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-systemd\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900239    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-sysconfig\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900239    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-run-ovn\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900260    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-var-lib-kubelet\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.900292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900289    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fae8e80b-baa0-4195-9657-aefc73c89147-cni-binary-copy\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900307    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-systemd\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900315    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900357    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900362    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-var-lib-kubelet\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900375    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-cni-bin\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900355    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-sysctl-d\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900397    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-modprobe-d\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900421    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-cni-bin\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900421    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-cni-dir\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.900446    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900474    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-system-cni-dir\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900513    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900551    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-modprobe-d\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900452    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-system-cni-dir\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.900587    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs podName:79b5c08b-176c-4374-a72b-97d488dcc046 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:36.400565496 +0000 UTC m=+3.163950817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs") pod "network-metrics-daemon-gr9j2" (UID: "79b5c08b-176c-4374-a72b-97d488dcc046") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900601    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-cni-dir\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.900915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900623    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fae8e80b-baa0-4195-9657-aefc73c89147-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900664    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-systemd-units\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900693    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900712    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-system-cni-dir\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900728    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-var-lib-cni-multus\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900731    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-systemd-units\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900746    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-daemon-config\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900767    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdvsn\" (UniqueName: \"kubernetes.io/projected/bde91108-8391-455f-ae1f-ed867f2e9bdf-kube-api-access-fdvsn\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900802    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900815    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26m22\" (UniqueName: \"kubernetes.io/projected/fae8e80b-baa0-4195-9657-aefc73c89147-kube-api-access-26m22\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900804    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-system-cni-dir\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900851    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-cni-netd\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900881    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dfb74e7-e835-4ed2-ab1d-55985f662845-ovnkube-script-lib\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900911    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-run-multus-certs\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900949    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-run\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900966    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fae8e80b-baa0-4195-9657-aefc73c89147-cni-binary-copy\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.900977    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-kubernetes\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.901780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901009    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a885d1ba-0164-40a1-a476-b982c984dbe7-agent-certs\") pod \"konnectivity-agent-wzk2v\" (UID: \"a885d1ba-0164-40a1-a476-b982c984dbe7\") " pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901048    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-cni-netd\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901066    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-run\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901018    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2dfb74e7-e835-4ed2-ab1d-55985f662845-ovnkube-config\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901121    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-kubernetes\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901104    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-run-multus-certs\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901132    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-var-lib-cni-multus\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901211    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fae8e80b-baa0-4195-9657-aefc73c89147-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901247    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgm7t\" (UniqueName: \"kubernetes.io/projected/79b5c08b-176c-4374-a72b-97d488dcc046-kube-api-access-bgm7t\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901299    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-run-netns\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901335    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-cnibin\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901364    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-run-netns\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901402    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-var-lib-cni-bin\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901408    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-daemon-config\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901433    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-cnibin\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901365    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-var-lib-cni-bin\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901532    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-var-lib-openvswitch\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.902660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901566    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-etc-openvswitch\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901567    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2dfb74e7-e835-4ed2-ab1d-55985f662845-ovnkube-script-lib\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901582    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dfb74e7-e835-4ed2-ab1d-55985f662845-env-overrides\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901612    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dfb74e7-e835-4ed2-ab1d-55985f662845-ovn-node-metrics-cert\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901592    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-var-lib-openvswitch\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901641    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjt9r\" (UniqueName: \"kubernetes.io/projected/2dfb74e7-e835-4ed2-ab1d-55985f662845-kube-api-access-sjt9r\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901654    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-etc-openvswitch\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901752    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpn9k\" (UniqueName: \"kubernetes.io/projected/9c7b1df8-a5d3-4087-9362-725f43f825d6-kube-api-access-zpn9k\") pod \"node-resolver-z2vnx\" (UID: \"9c7b1df8-a5d3-4087-9362-725f43f825d6\") " pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901790    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-log-socket\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901815    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-run-ovn-kubernetes\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901864    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-run-ovn-kubernetes\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901864    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-log-socket\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901886    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3e26069-93f9-43ce-9605-f1da44d37f09-dbus\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901918    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901945    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-os-release\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901971    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fae8e80b-baa0-4195-9657-aefc73c89147-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901997    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-sys\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.902003    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:35.903530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902023    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-sysctl-conf\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902042    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fae8e80b-baa0-4195-9657-aefc73c89147-os-release\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:35.902053    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret podName:e3e26069-93f9-43ce-9605-f1da44d37f09 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:36.402037155 +0000 UTC m=+3.165422458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret") pod "global-pull-secret-syncer-qzshm" (UID: "e3e26069-93f9-43ce-9605-f1da44d37f09") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902049    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-sys\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901998    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3e26069-93f9-43ce-9605-f1da44d37f09-dbus\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.901974    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2dfb74e7-e835-4ed2-ab1d-55985f662845-env-overrides\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902075    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-os-release\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902154    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-sysctl-conf\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902186    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztp4\" (UniqueName: \"kubernetes.io/projected/daa7d524-66f0-4937-83f2-8a7a0766dd9a-kube-api-access-zztp4\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902221    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a885d1ba-0164-40a1-a476-b982c984dbe7-konnectivity-ca\") pod \"konnectivity-agent-wzk2v\" (UID: \"a885d1ba-0164-40a1-a476-b982c984dbe7\") " pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902194    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-os-release\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902335    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-cnibin\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902367    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-run-k8s-cni-cncf-io\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902393    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-etc-kubernetes\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902405    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-cnibin\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902420    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-run-systemd\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902436    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-host-run-k8s-cni-cncf-io\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902455    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-etc-kubernetes\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902477    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-node-log\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902508    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-run-systemd\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902542    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-node-log\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902547    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-socket-dir-parent\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902578    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-hostroot\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902588    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fae8e80b-baa0-4195-9657-aefc73c89147-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902603    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3e26069-93f9-43ce-9605-f1da44d37f09-kubelet-config\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902603    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-multus-socket-dir-parent\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902627    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-lib-modules\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902648    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-host\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902671    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c7b1df8-a5d3-4087-9362-725f43f825d6-tmp-dir\") pod \"node-resolver-z2vnx\" (UID: \"9c7b1df8-a5d3-4087-9362-725f43f825d6\") " pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902629    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bde91108-8391-455f-ae1f-ed867f2e9bdf-hostroot\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902683    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3e26069-93f9-43ce-9605-f1da44d37f09-kubelet-config\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902700    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-slash\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902727    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-var-lib-kubelet\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902747    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-lib-modules\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902753    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/daa7d524-66f0-4937-83f2-8a7a0766dd9a-tmp\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.904810 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902766    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-host\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902775    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a885d1ba-0164-40a1-a476-b982c984dbe7-konnectivity-ca\") pod \"konnectivity-agent-wzk2v\" (UID: \"a885d1ba-0164-40a1-a476-b982c984dbe7\") " pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902846    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/daa7d524-66f0-4937-83f2-8a7a0766dd9a-var-lib-kubelet\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902882    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-host-slash\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902918    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bde91108-8391-455f-ae1f-ed867f2e9bdf-cni-binary-copy\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902947    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-tuned\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902972    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c7b1df8-a5d3-4087-9362-725f43f825d6-tmp-dir\") pod \"node-resolver-z2vnx\" (UID: \"9c7b1df8-a5d3-4087-9362-725f43f825d6\") " pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.902974    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c7b1df8-a5d3-4087-9362-725f43f825d6-hosts-file\") pod \"node-resolver-z2vnx\" (UID: \"9c7b1df8-a5d3-4087-9362-725f43f825d6\") " pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.903015    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9c7b1df8-a5d3-4087-9362-725f43f825d6-hosts-file\") pod \"node-resolver-z2vnx\" (UID: \"9c7b1df8-a5d3-4087-9362-725f43f825d6\") " pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.903031    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-run-openvswitch\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.903109    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2dfb74e7-e835-4ed2-ab1d-55985f662845-run-openvswitch\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.903426    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bde91108-8391-455f-ae1f-ed867f2e9bdf-cni-binary-copy\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.903852    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a885d1ba-0164-40a1-a476-b982c984dbe7-agent-certs\") pod \"konnectivity-agent-wzk2v\" (UID: \"a885d1ba-0164-40a1-a476-b982c984dbe7\") " pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.904189    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dfb74e7-e835-4ed2-ab1d-55985f662845-ovn-node-metrics-cert\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.904762    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/daa7d524-66f0-4937-83f2-8a7a0766dd9a-tmp\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.905363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.904944    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/daa7d524-66f0-4937-83f2-8a7a0766dd9a-etc-tuned\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.915622 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.915601    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdvsn\" (UniqueName: \"kubernetes.io/projected/bde91108-8391-455f-ae1f-ed867f2e9bdf-kube-api-access-fdvsn\") pod \"multus-n6zrx\" (UID: \"bde91108-8391-455f-ae1f-ed867f2e9bdf\") " pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:35.917463 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.917440    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztp4\" (UniqueName: \"kubernetes.io/projected/daa7d524-66f0-4937-83f2-8a7a0766dd9a-kube-api-access-zztp4\") pod \"tuned-rcn6r\" (UID: \"daa7d524-66f0-4937-83f2-8a7a0766dd9a\") " pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:35.917926 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.917907    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpn9k\" (UniqueName: \"kubernetes.io/projected/9c7b1df8-a5d3-4087-9362-725f43f825d6-kube-api-access-zpn9k\") pod \"node-resolver-z2vnx\" (UID: \"9c7b1df8-a5d3-4087-9362-725f43f825d6\") " pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:35.918028 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.918010    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgm7t\" (UniqueName: \"kubernetes.io/projected/79b5c08b-176c-4374-a72b-97d488dcc046-kube-api-access-bgm7t\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:35.919059 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.919039    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjt9r\" (UniqueName: \"kubernetes.io/projected/2dfb74e7-e835-4ed2-ab1d-55985f662845-kube-api-access-sjt9r\") pod \"ovnkube-node-6krdm\" (UID: \"2dfb74e7-e835-4ed2-ab1d-55985f662845\") " pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:35.919715 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.919698    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26m22\" (UniqueName: \"kubernetes.io/projected/fae8e80b-baa0-4195-9657-aefc73c89147-kube-api-access-26m22\") pod \"multus-additional-cni-plugins-jn8cz\" (UID: \"fae8e80b-baa0-4195-9657-aefc73c89147\") " pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:35.986282 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.986223    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4f9jp"
Jul 02 14:16:35.994271 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:35.994247    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8qtb7"
Jul 02 14:16:35.997185 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:35.997159    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf27d440_b641_4efc_9a92_273c9bbef679.slice/crio-163d7b9233cc329ede974c93a6dbd8ba0f96fcfa82453c395bb0b2f8be679755 WatchSource:0}: Error finding container 163d7b9233cc329ede974c93a6dbd8ba0f96fcfa82453c395bb0b2f8be679755: Status 404 returned error can't find the container with id 163d7b9233cc329ede974c93a6dbd8ba0f96fcfa82453c395bb0b2f8be679755
Jul 02 14:16:36.002780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.001868    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j"
Jul 02 14:16:36.005154 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:36.005134    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29ddb9d_0592_481b_96f9_90acc879de60.slice/crio-11960e9070964baa9ed2a730ea7abdf678379f1bb41bc3e05799637b74502d07 WatchSource:0}: Error finding container 11960e9070964baa9ed2a730ea7abdf678379f1bb41bc3e05799637b74502d07: Status 404 returned error can't find the container with id 11960e9070964baa9ed2a730ea7abdf678379f1bb41bc3e05799637b74502d07
Jul 02 14:16:36.006597 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.006573    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcn6r"
Jul 02 14:16:36.012424 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.012403    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z2vnx"
Jul 02 14:16:36.012742 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:36.012719    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0399b6ab_86be_4b3b_bbc7_2bc4a46a05e3.slice/crio-664df8d270640c9847aeefa5f3af476a303938a1b6f3cdccf56a74b5adb39ee3 WatchSource:0}: Error finding container 664df8d270640c9847aeefa5f3af476a303938a1b6f3cdccf56a74b5adb39ee3: Status 404 returned error can't find the container with id 664df8d270640c9847aeefa5f3af476a303938a1b6f3cdccf56a74b5adb39ee3
Jul 02 14:16:36.017084 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:36.017064    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa7d524_66f0_4937_83f2_8a7a0766dd9a.slice/crio-22adcf238d2aadbf7e92754c5d7c65c52946b108a568453fc1e82998fe55adf3 WatchSource:0}: Error finding container 22adcf238d2aadbf7e92754c5d7c65c52946b108a568453fc1e82998fe55adf3: Status 404 returned error can't find the container with id 22adcf238d2aadbf7e92754c5d7c65c52946b108a568453fc1e82998fe55adf3
Jul 02 14:16:36.018404 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.018368    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:16:36.022308 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:36.022286    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7b1df8_a5d3_4087_9362_725f43f825d6.slice/crio-71d1914715a5319275ac4330e8df5856469108f5fb2d4d0721781ae98e2475f4 WatchSource:0}: Error finding container 71d1914715a5319275ac4330e8df5856469108f5fb2d4d0721781ae98e2475f4: Status 404 returned error can't find the container with id 71d1914715a5319275ac4330e8df5856469108f5fb2d4d0721781ae98e2475f4
Jul 02 14:16:36.024857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.024836    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:36.028053 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:36.028029    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfb74e7_e835_4ed2_ab1d_55985f662845.slice/crio-495a525e74aad00162279896d78337b0e019f6514c60fa98b24fb4c04314a896 WatchSource:0}: Error finding container 495a525e74aad00162279896d78337b0e019f6514c60fa98b24fb4c04314a896: Status 404 returned error can't find the container with id 495a525e74aad00162279896d78337b0e019f6514c60fa98b24fb4c04314a896
Jul 02 14:16:36.030562 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.030525    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n6zrx"
Jul 02 14:16:36.033976 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:36.033953    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda885d1ba_0164_40a1_a476_b982c984dbe7.slice/crio-87e538520df8d9608df8766afa1a15502cb8e86aab3f8f783fcb1fc55b7b3158 WatchSource:0}: Error finding container 87e538520df8d9608df8766afa1a15502cb8e86aab3f8f783fcb1fc55b7b3158: Status 404 returned error can't find the container with id 87e538520df8d9608df8766afa1a15502cb8e86aab3f8f783fcb1fc55b7b3158
Jul 02 14:16:36.034817 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.034800    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jn8cz"
Jul 02 14:16:36.039521 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:36.039502    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde91108_8391_455f_ae1f_ed867f2e9bdf.slice/crio-9ca0ab89091196d9c0929297f6136c8617589170c244b298e3128e909ef2281d WatchSource:0}: Error finding container 9ca0ab89091196d9c0929297f6136c8617589170c244b298e3128e909ef2281d: Status 404 returned error can't find the container with id 9ca0ab89091196d9c0929297f6136c8617589170c244b298e3128e909ef2281d
Jul 02 14:16:36.044830 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:16:36.044809    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae8e80b_baa0_4195_9657_aefc73c89147.slice/crio-1904af0b3bb48695eda437bb2e1afb384cdce8df5e142d16eb8582142ac9df1d WatchSource:0}: Error finding container 1904af0b3bb48695eda437bb2e1afb384cdce8df5e142d16eb8582142ac9df1d: Status 404 returned error can't find the container with id 1904af0b3bb48695eda437bb2e1afb384cdce8df5e142d16eb8582142ac9df1d
Jul 02 14:16:36.406975 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.406946    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:36.407168 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.406988    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:36.407168 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:36.407154    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:16:36.407291 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:36.407184    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:16:36.407291 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.407187    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:36.407291 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:36.407199    2551 projected.go:196] Error preparing data for projected volume kube-api-access-65r9n for pod openshift-network-diagnostics/network-check-target-jpqgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:36.407291 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:36.407166    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:36.407291 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:36.407262    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:36.407291 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:36.407268    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n podName:065c8aea-2847-43cd-bff7-ba1e706a032d nodeName:}" failed. No retries permitted until 2026-07-02 14:16:37.4072463 +0000 UTC m=+4.170631629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-65r9n" (UniqueName: "kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n") pod "network-check-target-jpqgc" (UID: "065c8aea-2847-43cd-bff7-ba1e706a032d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:36.407706 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:36.407303    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs podName:79b5c08b-176c-4374-a72b-97d488dcc046 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:37.407282673 +0000 UTC m=+4.170667990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs") pod "network-metrics-daemon-gr9j2" (UID: "79b5c08b-176c-4374-a72b-97d488dcc046") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:36.407706 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:36.407322    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret podName:e3e26069-93f9-43ce-9605-f1da44d37f09 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:37.407312487 +0000 UTC m=+4.170697790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret") pod "global-pull-secret-syncer-qzshm" (UID: "e3e26069-93f9-43ce-9605-f1da44d37f09") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:36.763459 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.763112    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerStarted","Data":"1904af0b3bb48695eda437bb2e1afb384cdce8df5e142d16eb8582142ac9df1d"}
Jul 02 14:16:36.767731 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.767701    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n6zrx" event={"ID":"bde91108-8391-455f-ae1f-ed867f2e9bdf","Type":"ContainerStarted","Data":"9ca0ab89091196d9c0929297f6136c8617589170c244b298e3128e909ef2281d"}
Jul 02 14:16:36.781780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.781714    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wzk2v" event={"ID":"a885d1ba-0164-40a1-a476-b982c984dbe7","Type":"ContainerStarted","Data":"87e538520df8d9608df8766afa1a15502cb8e86aab3f8f783fcb1fc55b7b3158"}
Jul 02 14:16:36.796289 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.796260    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerStarted","Data":"495a525e74aad00162279896d78337b0e019f6514c60fa98b24fb4c04314a896"}
Jul 02 14:16:36.799597 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.799569    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z2vnx" event={"ID":"9c7b1df8-a5d3-4087-9362-725f43f825d6","Type":"ContainerStarted","Data":"71d1914715a5319275ac4330e8df5856469108f5fb2d4d0721781ae98e2475f4"}
Jul 02 14:16:36.807343 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.807315    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j" event={"ID":"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3","Type":"ContainerStarted","Data":"664df8d270640c9847aeefa5f3af476a303938a1b6f3cdccf56a74b5adb39ee3"}
Jul 02 14:16:36.816069 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.816041    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8qtb7" event={"ID":"d29ddb9d-0592-481b-96f9-90acc879de60","Type":"ContainerStarted","Data":"11960e9070964baa9ed2a730ea7abdf678379f1bb41bc3e05799637b74502d07"}
Jul 02 14:16:36.820816 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.820787    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcn6r" event={"ID":"daa7d524-66f0-4937-83f2-8a7a0766dd9a","Type":"ContainerStarted","Data":"22adcf238d2aadbf7e92754c5d7c65c52946b108a568453fc1e82998fe55adf3"}
Jul 02 14:16:36.820923 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.820899    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 14:11:36 +0000 UTC" deadline="2027-12-01 08:25:52.717060342 +0000 UTC"
Jul 02 14:16:36.820989 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.820921    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12402h9m15.89614335s"
Jul 02 14:16:36.832219 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.832190    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4f9jp" event={"ID":"bf27d440-b641-4efc-9a92-273c9bbef679","Type":"ContainerStarted","Data":"163d7b9233cc329ede974c93a6dbd8ba0f96fcfa82453c395bb0b2f8be679755"}
Jul 02 14:16:36.835562 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:36.835523    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal" event={"ID":"5e2f6dd71485cba712a5a6213acf7cad","Type":"ContainerStarted","Data":"c18871a162d1fcbc2ef09315dd9ea8ef236abf1f48e5a6d442907bb1bb659dc3"}
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:37.414822    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:37.414892    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:37.414938    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.415089    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.415112    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.415125    2551 projected.go:196] Error preparing data for projected volume kube-api-access-65r9n for pod openshift-network-diagnostics/network-check-target-jpqgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.415208    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n podName:065c8aea-2847-43cd-bff7-ba1e706a032d nodeName:}" failed. No retries permitted until 2026-07-02 14:16:39.415188597 +0000 UTC m=+6.178573901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-65r9n" (UniqueName: "kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n") pod "network-check-target-jpqgc" (UID: "065c8aea-2847-43cd-bff7-ba1e706a032d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.415644    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.415696    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs podName:79b5c08b-176c-4374-a72b-97d488dcc046 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:39.415679358 +0000 UTC m=+6.179064675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs") pod "network-metrics-daemon-gr9j2" (UID: "79b5c08b-176c-4374-a72b-97d488dcc046") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.415752    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:37.415835 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.415787    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret podName:e3e26069-93f9-43ce-9605-f1da44d37f09 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:39.41577573 +0000 UTC m=+6.179161035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret") pod "global-pull-secret-syncer-qzshm" (UID: "e3e26069-93f9-43ce-9605-f1da44d37f09") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:37.744706 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:37.744626    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:37.744878 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:37.744631    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:37.744878 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.744781    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:37.744878 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:37.744804    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:37.745055 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.744908    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:37.745055 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:37.744987    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:37.821427 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:37.821378    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 14:11:36 +0000 UTC" deadline="2028-02-19 21:30:29.769464818 +0000 UTC"
Jul 02 14:16:37.821427 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:37.821423    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14335h13m51.948045666s"
Jul 02 14:16:39.433661 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:39.433463    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:39.433661 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:39.433557    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:39.433661 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:39.433600    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:39.434224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.433754    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:39.434224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.433776    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:16:39.434224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.433806    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:16:39.434224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.433809    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:39.434224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.433820    2551 projected.go:196] Error preparing data for projected volume kube-api-access-65r9n for pod openshift-network-diagnostics/network-check-target-jpqgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:39.434224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.433826    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret podName:e3e26069-93f9-43ce-9605-f1da44d37f09 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:43.433805267 +0000 UTC m=+10.197190583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret") pod "global-pull-secret-syncer-qzshm" (UID: "e3e26069-93f9-43ce-9605-f1da44d37f09") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:39.434224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.433848    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs podName:79b5c08b-176c-4374-a72b-97d488dcc046 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:43.433837857 +0000 UTC m=+10.197223173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs") pod "network-metrics-daemon-gr9j2" (UID: "79b5c08b-176c-4374-a72b-97d488dcc046") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:39.434224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.433871    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n podName:065c8aea-2847-43cd-bff7-ba1e706a032d nodeName:}" failed. No retries permitted until 2026-07-02 14:16:43.433855614 +0000 UTC m=+10.197240930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-65r9n" (UniqueName: "kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n") pod "network-check-target-jpqgc" (UID: "065c8aea-2847-43cd-bff7-ba1e706a032d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:39.747628 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:39.747501    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:39.747788 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:39.747512    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:39.747788 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.747727    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:39.747788 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:39.747766    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:39.748213 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.748184    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:39.748422 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:39.747900    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:41.743699 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:41.743645    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:41.744144 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:41.743780    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:41.744144 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:41.743650    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:41.744144 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:41.743951    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:41.744144 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:41.744013    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:41.744144 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:41.744082    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:43.468079 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:43.467861    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:43.468079 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:43.467923    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:43.468079 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:43.467965    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:43.468079 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.468071    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:43.468686 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.468111    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:16:43.468686 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.468127    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:16:43.468686 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.468139    2551 projected.go:196] Error preparing data for projected volume kube-api-access-65r9n for pod openshift-network-diagnostics/network-check-target-jpqgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:43.468686 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.468142    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs podName:79b5c08b-176c-4374-a72b-97d488dcc046 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:51.468121234 +0000 UTC m=+18.231506537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs") pod "network-metrics-daemon-gr9j2" (UID: "79b5c08b-176c-4374-a72b-97d488dcc046") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:43.468686 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.468187    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n podName:065c8aea-2847-43cd-bff7-ba1e706a032d nodeName:}" failed. No retries permitted until 2026-07-02 14:16:51.468172624 +0000 UTC m=+18.231557937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-65r9n" (UniqueName: "kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n") pod "network-check-target-jpqgc" (UID: "065c8aea-2847-43cd-bff7-ba1e706a032d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:43.468686 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.468255    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:43.468686 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.468294    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret podName:e3e26069-93f9-43ce-9605-f1da44d37f09 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:51.468282191 +0000 UTC m=+18.231667496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret") pod "global-pull-secret-syncer-qzshm" (UID: "e3e26069-93f9-43ce-9605-f1da44d37f09") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:43.744258 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:43.744181    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:43.744258 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:43.744185    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:43.745286 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:43.745243    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:43.745435 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.745400    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:43.745663 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.745539    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:43.745663 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:43.745623    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:45.744572 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:45.744312    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:45.744998 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:45.744309    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:45.744998 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:45.744309    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:45.744998 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:45.744697    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:45.744998 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:45.744768    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:45.744998 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:45.744842    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:47.743555 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:47.743529    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:47.743555 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:47.743549    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:47.743910 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:47.743530    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:47.743910 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:47.743620    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:47.743910 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:47.743695    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:47.743910 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:47.743768    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:49.746868 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:49.746837    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:49.747300 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:49.746953    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:49.747300 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:49.746995    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:49.747300 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:49.747011    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:49.747300 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:49.747102    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:49.747300 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:49.747207    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:51.526420 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:51.526382    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:51.526442    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:51.526505    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.526569    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.526595    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.526610    2551 projected.go:196] Error preparing data for projected volume kube-api-access-65r9n for pod openshift-network-diagnostics/network-check-target-jpqgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.526616    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.526576    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.526657    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n podName:065c8aea-2847-43cd-bff7-ba1e706a032d nodeName:}" failed. No retries permitted until 2026-07-02 14:17:07.526644919 +0000 UTC m=+34.290030218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-65r9n" (UniqueName: "kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n") pod "network-check-target-jpqgc" (UID: "065c8aea-2847-43cd-bff7-ba1e706a032d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.526671    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret podName:e3e26069-93f9-43ce-9605-f1da44d37f09 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:07.526665502 +0000 UTC m=+34.290050800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret") pod "global-pull-secret-syncer-qzshm" (UID: "e3e26069-93f9-43ce-9605-f1da44d37f09") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:51.526899 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.526681    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs podName:79b5c08b-176c-4374-a72b-97d488dcc046 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:07.526675489 +0000 UTC m=+34.290060787 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs") pod "network-metrics-daemon-gr9j2" (UID: "79b5c08b-176c-4374-a72b-97d488dcc046") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:51.744320 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:51.744289    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:51.744522 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:51.744289    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:51.744522 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.744413    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:51.744522 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:51.744441    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:51.744695 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.744540    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:51.744695 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:51.744612    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:53.746107 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:53.746075    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:53.746553 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:53.746078    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:53.746553 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:53.746167    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:53.746553 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:53.746078    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:53.746553 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:53.746274    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:53.746553 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:53.746323    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:55.743852 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:55.743823    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:55.743852 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:55.743838    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:55.744224 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:55.743823    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:55.744224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:55.743916    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:55.744224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:55.744010    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:55.744224 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:55.744103    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:56.889621 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.889195    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal" event={"ID":"b9e4c4239276a54ecd642935c5ffe6cd","Type":"ContainerStarted","Data":"c5fd3f40f32b3865de2061d1a6c57d42ce0268658f851c1d0e94064bf13da8d1"}
Jul 02 14:16:56.890687 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.890657    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerStarted","Data":"9e898ffaccaf88ed1cc0df0846466d8a0708339c5caee9545a8551d137bf0a86"}
Jul 02 14:16:56.892180 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.892158    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n6zrx" event={"ID":"bde91108-8391-455f-ae1f-ed867f2e9bdf","Type":"ContainerStarted","Data":"dfd827cb01c74d364d3944f4ce52e7955a3a70556ccb3bc3e4f8a1975c113258"}
Jul 02 14:16:56.893521 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.893496    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wzk2v" event={"ID":"a885d1ba-0164-40a1-a476-b982c984dbe7","Type":"ContainerStarted","Data":"196d4d7f6b2f98bd97839517fa462e77144b91fa683f62b62e18c1838bcf784f"}
Jul 02 14:16:56.896332 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.896314    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:16:56.896683 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.896660    2551 generic.go:356] "Generic (PLEG): container finished" podID="2dfb74e7-e835-4ed2-ab1d-55985f662845" containerID="95a5a0304985610291f736c7e5c81ef55afcf4a2848efe8f469839d075ad166e" exitCode=1
Jul 02 14:16:56.896748 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.896730    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerStarted","Data":"ddad1bacf47864084e1c5af1d33b7dc6e81189b88e508cfc453196e94e35ad3d"}
Jul 02 14:16:56.896800 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.896787    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerStarted","Data":"191e40ea78564ba17176441cbc6c7e8c991318a4074839066a759a6ad6334e94"}
Jul 02 14:16:56.896834 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.896821    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerStarted","Data":"044c269bde7c5ad727eaeca74c61830c582aa6588fca42a9f0050c57e96d2aba"}
Jul 02 14:16:56.896874 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.896862    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerStarted","Data":"cbe266ec6ae3981ebdb98e873acabca64f72c9b43c5f1372448c03d472b02f0c"}
Jul 02 14:16:56.896912 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.896906    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerDied","Data":"95a5a0304985610291f736c7e5c81ef55afcf4a2848efe8f469839d075ad166e"}
Jul 02 14:16:56.896968 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.896955    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerStarted","Data":"990b3bab0b5d4172c33749f20c63a2ff4202a04073723a6dceb7d64100f7ae99"}
Jul 02 14:16:56.898169 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.898148    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z2vnx" event={"ID":"9c7b1df8-a5d3-4087-9362-725f43f825d6","Type":"ContainerStarted","Data":"9b6484674c07bc8d08431de87ba77a6eb4031e8e56adaa69df5cc3da80365e46"}
Jul 02 14:16:56.899455 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.899436    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j" event={"ID":"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3","Type":"ContainerStarted","Data":"277b30842f114948eb8588ca37eb36978dc9f9a33cfb215c327af313e557fece"}
Jul 02 14:16:56.900704 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.900688    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcn6r" event={"ID":"daa7d524-66f0-4937-83f2-8a7a0766dd9a","Type":"ContainerStarted","Data":"919e2ea08f386b6707a8d5868701aa8543bad64b3a07a4f8761ef294d9852a47"}
Jul 02 14:16:56.901983 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.901966    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4f9jp" event={"ID":"bf27d440-b641-4efc-9a92-273c9bbef679","Type":"ContainerStarted","Data":"9e4ecaef1547b47f5f98c5a32cd746f7338bd1c569151f8db3fcf1661bef7474"}
Jul 02 14:16:56.903262 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.903245    2551 generic.go:356] "Generic (PLEG): container finished" podID="5e2f6dd71485cba712a5a6213acf7cad" containerID="a7392e44417c7f38757c0d3b4493cfae230b02e6aa4128ed43ad01ce0b18507d" exitCode=0
Jul 02 14:16:56.903320 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.903270    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal" event={"ID":"5e2f6dd71485cba712a5a6213acf7cad","Type":"ContainerDied","Data":"a7392e44417c7f38757c0d3b4493cfae230b02e6aa4128ed43ad01ce0b18507d"}
Jul 02 14:16:56.914779 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.914740    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-54.ec2.internal" podStartSLOduration=21.914729235 podStartE2EDuration="21.914729235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:56.914179125 +0000 UTC m=+23.677564447" watchObservedRunningTime="2026-07-02 14:16:56.914729235 +0000 UTC m=+23.678114553"
Jul 02 14:16:56.938509 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.938446    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4f9jp" podStartSLOduration=2.967013993 podStartE2EDuration="22.938434287s" totalImagesPullingTime="19.971420294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:56.93823534 +0000 UTC m=+23.701620662" watchObservedRunningTime="2026-07-02 14:16:56.938434287 +0000 UTC m=+23.701819608"
Jul 02 14:16:56.987418 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.987378    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rcn6r" podStartSLOduration=3.014737567 podStartE2EDuration="22.987367784s" totalImagesPullingTime="19.972630217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:56.987054633 +0000 UTC m=+23.750439952" watchObservedRunningTime="2026-07-02 14:16:56.987367784 +0000 UTC m=+23.750753104"
Jul 02 14:16:56.987563 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:56.987505    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z2vnx" podStartSLOduration=3.038797747 podStartE2EDuration="22.987497354s" totalImagesPullingTime="19.948699607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:56.962833708 +0000 UTC m=+23.726219028" watchObservedRunningTime="2026-07-02 14:16:56.987497354 +0000 UTC m=+23.750882673"
Jul 02 14:16:57.010172 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:57.010135    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wzk2v" podStartSLOduration=3.072825238 podStartE2EDuration="23.010123773s" totalImagesPullingTime="19.937298535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:57.00972595 +0000 UTC m=+23.773111272" watchObservedRunningTime="2026-07-02 14:16:57.010123773 +0000 UTC m=+23.773509090"
Jul 02 14:16:57.033688 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:57.033608    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n6zrx" podStartSLOduration=3.068158656 podStartE2EDuration="23.03359751s" totalImagesPullingTime="19.965438854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:57.033529212 +0000 UTC m=+23.796914533" watchObservedRunningTime="2026-07-02 14:16:57.03359751 +0000 UTC m=+23.796982830"
Jul 02 14:16:57.744287 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:57.744206    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:57.744843 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:57.744206    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:57.744843 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:57.744323    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:57.744843 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:57.744211    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:57.744843 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:57.744417    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:57.744843 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:57.744517    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:57.906994 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:57.906959    2551 generic.go:356] "Generic (PLEG): container finished" podID="fae8e80b-baa0-4195-9657-aefc73c89147" containerID="9e898ffaccaf88ed1cc0df0846466d8a0708339c5caee9545a8551d137bf0a86" exitCode=0
Jul 02 14:16:57.907426 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:57.907000    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerDied","Data":"9e898ffaccaf88ed1cc0df0846466d8a0708339c5caee9545a8551d137bf0a86"}
Jul 02 14:16:57.908928 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:57.908871    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8qtb7" event={"ID":"d29ddb9d-0592-481b-96f9-90acc879de60","Type":"ContainerStarted","Data":"33f20ee5dabf658fed972401fdb5e3105217d40143001b943b50bd96a41d2b15"}
Jul 02 14:16:57.974012 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:57.973461    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8qtb7" podStartSLOduration=4.007517177 podStartE2EDuration="23.973316473s" totalImagesPullingTime="19.965799296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:57.973177612 +0000 UTC m=+24.736562933" watchObservedRunningTime="2026-07-02 14:16:57.973316473 +0000 UTC m=+24.736701795"
Jul 02 14:16:58.337882 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:58.337859    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 14:16:58.761637 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:58.761534    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T14:16:58.337878424Z","UUID":"3096773b-7942-4732-870e-3ce6c24b52e9","Handler":null,"Name":"","Endpoint":""}
Jul 02 14:16:58.763401 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:58.763381    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 14:16:58.763533 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:58.763427    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 14:16:58.914723 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:58.914565    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal" event={"ID":"5e2f6dd71485cba712a5a6213acf7cad","Type":"ContainerStarted","Data":"ab812a000f8c223b601984b35aed53208ed8c0c02ba571cf26c9cb2a8e254bed"}
Jul 02 14:16:58.919434 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:58.919415    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:16:58.926027 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:58.925965    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j" event={"ID":"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3","Type":"ContainerStarted","Data":"235ce29d8ea06047ace4dc0ba7662b1161c011e3e5badba82819807a6c4f0f0a"}
Jul 02 14:16:58.932562 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:58.932518    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-54.ec2.internal" podStartSLOduration=23.932502968 podStartE2EDuration="23.932502968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:58.931974498 +0000 UTC m=+25.695359820" watchObservedRunningTime="2026-07-02 14:16:58.932502968 +0000 UTC m=+25.695888289"
Jul 02 14:16:59.092527 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.092434    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:59.093303 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.093279    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:59.744387 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.744318    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:16:59.744561 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.744318    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:16:59.744561 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:59.744434    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:16:59.744561 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.744319    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:16:59.744725 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:59.744552    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:16:59.744725 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:16:59.744649    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:16:59.931924 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.931894    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:16:59.932360 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.932326    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerStarted","Data":"7d2919ce6ac3bb8ee148b51f5654abc2cbb5e33bf757195702a88554efab5034"}
Jul 02 14:16:59.934665 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.934600    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j" event={"ID":"0399b6ab-86be-4b3b-bbc7-2bc4a46a05e3","Type":"ContainerStarted","Data":"7c330019666432fba0b0dc7a894fd297acc6c07f21701831b91cdfe34612207a"}
Jul 02 14:16:59.935293 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.935045    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:59.935436 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.935423    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wzk2v"
Jul 02 14:16:59.954806 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:16:59.954764    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rcw4j" podStartSLOduration=3.6933078679999998 podStartE2EDuration="25.954749892s" totalImagesPullingTime="22.261442024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:59.954313616 +0000 UTC m=+26.717698937" watchObservedRunningTime="2026-07-02 14:16:59.954749892 +0000 UTC m=+26.718135210"
Jul 02 14:17:01.743783 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:01.743702    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:17:01.744206 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:01.743713    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:01.744206 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:01.743826    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:17:01.744206 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:01.743702    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:01.744206 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:01.743873    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:17:01.744206 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:01.743942    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:17:02.942532 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:02.942264    2551 generic.go:356] "Generic (PLEG): container finished" podID="fae8e80b-baa0-4195-9657-aefc73c89147" containerID="032600f0fe79a31543c87262554e9dca57ca6de33f5462c0a37ce7559942f6ba" exitCode=0
Jul 02 14:17:02.943346 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:02.942328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerDied","Data":"032600f0fe79a31543c87262554e9dca57ca6de33f5462c0a37ce7559942f6ba"}
Jul 02 14:17:02.945994 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:02.945980    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:17:02.946275 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:02.946255    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerStarted","Data":"f845e8fe0b42f830bdb1ab2a3ea6ce0a0d5c16880820f5b64398e935df2eb3f6"}
Jul 02 14:17:02.946574 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:02.946555    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:17:02.946664 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:02.946633    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:17:02.946732 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:02.946715    2551 scope.go:117] "RemoveContainer" containerID="95a5a0304985610291f736c7e5c81ef55afcf4a2848efe8f469839d075ad166e"
Jul 02 14:17:02.961744 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:02.961724    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:17:03.744155 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:03.743947    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:17:03.744373 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:03.744020    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:03.744373 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:03.744263    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:17:03.744373 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:03.744046    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:03.744373 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:03.744319    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:17:03.744544 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:03.744373    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:17:03.954203 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:03.954175    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:17:03.954729 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:03.954701    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" event={"ID":"2dfb74e7-e835-4ed2-ab1d-55985f662845","Type":"ContainerStarted","Data":"4ab8dbbe70473b27ee30d330436955cb4944332bf3b7ccb0fe036a3fede25f87"}
Jul 02 14:17:03.954916 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:03.954897    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:17:03.969281 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:03.969258    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:17:03.983658 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:03.983615    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm" podStartSLOduration=9.79304485 podStartE2EDuration="29.983601895s" totalImagesPullingTime="20.190557045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:03.982075824 +0000 UTC m=+30.745461145" watchObservedRunningTime="2026-07-02 14:17:03.983601895 +0000 UTC m=+30.746987226"
Jul 02 14:17:04.173623 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:04.173589    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qzshm"]
Jul 02 14:17:04.173791 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:04.173752    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:04.173909 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:04.173885    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:17:04.176772 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:04.176742    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jpqgc"]
Jul 02 14:17:04.176916 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:04.176848    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:04.176975 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:04.176941    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:17:04.177447 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:04.177417    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gr9j2"]
Jul 02 14:17:04.177552 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:04.177532    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:17:04.177663 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:04.177642    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:17:05.743659 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:05.743588    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:17:05.743659 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:05.743604    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:05.744137 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:05.743590    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:05.744137 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:05.743695    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:17:05.744137 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:05.743741    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:17:05.744137 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:05.743797    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:17:05.961372 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:05.961343    2551 generic.go:356] "Generic (PLEG): container finished" podID="fae8e80b-baa0-4195-9657-aefc73c89147" containerID="4448f7d23f8da70dc4222ddf905ed590e00aa8f1e5c94754869de69432765927" exitCode=0
Jul 02 14:17:05.961526 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:05.961430    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerDied","Data":"4448f7d23f8da70dc4222ddf905ed590e00aa8f1e5c94754869de69432765927"}
Jul 02 14:17:07.546213 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:07.546186    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:07.546224    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:07.546253    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.546339    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.546345    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.546370    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.546380    2551 projected.go:196] Error preparing data for projected volume kube-api-access-65r9n for pod openshift-network-diagnostics/network-check-target-jpqgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.546345    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.546412    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs podName:79b5c08b-176c-4374-a72b-97d488dcc046 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:39.546400353 +0000 UTC m=+66.309785656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs") pod "network-metrics-daemon-gr9j2" (UID: "79b5c08b-176c-4374-a72b-97d488dcc046") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.546426    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n podName:065c8aea-2847-43cd-bff7-ba1e706a032d nodeName:}" failed. No retries permitted until 2026-07-02 14:17:39.546419477 +0000 UTC m=+66.309804777 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-65r9n" (UniqueName: "kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n") pod "network-check-target-jpqgc" (UID: "065c8aea-2847-43cd-bff7-ba1e706a032d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:17:07.546565 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.546438    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret podName:e3e26069-93f9-43ce-9605-f1da44d37f09 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:39.546431508 +0000 UTC m=+66.309816806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret") pod "global-pull-secret-syncer-qzshm" (UID: "e3e26069-93f9-43ce-9605-f1da44d37f09") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:17:07.743698 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:07.743551    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:07.743806 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:07.743551    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:17:07.743806 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.743773    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzshm" podUID="e3e26069-93f9-43ce-9605-f1da44d37f09"
Jul 02 14:17:07.743888 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:07.743627    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:07.743937 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.743887    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr9j2" podUID="79b5c08b-176c-4374-a72b-97d488dcc046"
Jul 02 14:17:07.744018 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:07.743997    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jpqgc" podUID="065c8aea-2847-43cd-bff7-ba1e706a032d"
Jul 02 14:17:07.967552 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:07.967485    2551 generic.go:356] "Generic (PLEG): container finished" podID="fae8e80b-baa0-4195-9657-aefc73c89147" containerID="ca74d79b18656054538dbcc38bd750912ed7408b8507cdff54439bb83d5cc926" exitCode=0
Jul 02 14:17:07.967552 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:07.967546    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerDied","Data":"ca74d79b18656054538dbcc38bd750912ed7408b8507cdff54439bb83d5cc926"}
Jul 02 14:17:08.025080 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.025058    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-54.ec2.internal" event="NodeReady"
Jul 02 14:17:08.025184 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.025176    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 14:17:08.093961 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.093939    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s2zwr"]
Jul 02 14:17:08.114796 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.114777    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kbvnk"]
Jul 02 14:17:08.114903 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.114889    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s2zwr"
Jul 02 14:17:08.120897 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.120680    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 14:17:08.121020 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.121010    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 14:17:08.123740 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.121441    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z4mlq\""
Jul 02 14:17:08.123740 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.121514    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 14:17:08.130746 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.130727    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s2zwr"]
Jul 02 14:17:08.130825 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.130755    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kbvnk"]
Jul 02 14:17:08.130825 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.130820    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.133624 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.133609    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mttc5\""
Jul 02 14:17:08.133943 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.133928    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 14:17:08.134013 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.133932    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 14:17:08.252158 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.252138    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.252258 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.252178    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-config-volume\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.252258 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.252197    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnkmm\" (UniqueName: \"kubernetes.io/projected/a92336ee-ed07-4194-9236-03c1c360edbd-kube-api-access-gnkmm\") pod \"ingress-canary-s2zwr\" (UID: \"a92336ee-ed07-4194-9236-03c1c360edbd\") " pod="openshift-ingress-canary/ingress-canary-s2zwr"
Jul 02 14:17:08.252369 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.252283    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-tmp-dir\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.252369 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.252332    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bls\" (UniqueName: \"kubernetes.io/projected/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-kube-api-access-68bls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.252452 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.252391    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a92336ee-ed07-4194-9236-03c1c360edbd-cert\") pod \"ingress-canary-s2zwr\" (UID: \"a92336ee-ed07-4194-9236-03c1c360edbd\") " pod="openshift-ingress-canary/ingress-canary-s2zwr"
Jul 02 14:17:08.353492 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.353443    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a92336ee-ed07-4194-9236-03c1c360edbd-cert\") pod \"ingress-canary-s2zwr\" (UID: \"a92336ee-ed07-4194-9236-03c1c360edbd\") " pod="openshift-ingress-canary/ingress-canary-s2zwr"
Jul 02 14:17:08.353593 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.353493    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.353593 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.353536    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-config-volume\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.353593 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.353560    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnkmm\" (UniqueName: \"kubernetes.io/projected/a92336ee-ed07-4194-9236-03c1c360edbd-kube-api-access-gnkmm\") pod \"ingress-canary-s2zwr\" (UID: \"a92336ee-ed07-4194-9236-03c1c360edbd\") " pod="openshift-ingress-canary/ingress-canary-s2zwr"
Jul 02 14:17:08.353726 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.353593    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-tmp-dir\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.353726 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.353629    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bls\" (UniqueName: \"kubernetes.io/projected/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-kube-api-access-68bls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.353726 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:08.353644    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:17:08.353726 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:08.353717    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls podName:0a8a8c4d-b364-4613-a6a1-7767c76a3efd nodeName:}" failed. No retries permitted until 2026-07-02 14:17:08.853698394 +0000 UTC m=+35.617083694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls") pod "dns-default-kbvnk" (UID: "0a8a8c4d-b364-4613-a6a1-7767c76a3efd") : secret "dns-default-metrics-tls" not found
Jul 02 14:17:08.353936 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.353920    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-tmp-dir\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.354054 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.354038    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-config-volume\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.357543 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.357527    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a92336ee-ed07-4194-9236-03c1c360edbd-cert\") pod \"ingress-canary-s2zwr\" (UID: \"a92336ee-ed07-4194-9236-03c1c360edbd\") " pod="openshift-ingress-canary/ingress-canary-s2zwr"
Jul 02 14:17:08.364750 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.364725    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bls\" (UniqueName: \"kubernetes.io/projected/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-kube-api-access-68bls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.364822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.364802    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnkmm\" (UniqueName: \"kubernetes.io/projected/a92336ee-ed07-4194-9236-03c1c360edbd-kube-api-access-gnkmm\") pod \"ingress-canary-s2zwr\" (UID: \"a92336ee-ed07-4194-9236-03c1c360edbd\") " pod="openshift-ingress-canary/ingress-canary-s2zwr"
Jul 02 14:17:08.425639 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.425620    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s2zwr"
Jul 02 14:17:08.634442 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.634288    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s2zwr"]
Jul 02 14:17:08.637257 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:17:08.637231    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92336ee_ed07_4194_9236_03c1c360edbd.slice/crio-82e58668ec6df3b83980f5a0c1a9cbc37136a909936a02328bcdf74eadd5e226 WatchSource:0}: Error finding container 82e58668ec6df3b83980f5a0c1a9cbc37136a909936a02328bcdf74eadd5e226: Status 404 returned error can't find the container with id 82e58668ec6df3b83980f5a0c1a9cbc37136a909936a02328bcdf74eadd5e226
Jul 02 14:17:08.856576 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.856504    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:08.856698 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:08.856626    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:17:08.856698 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:08.856678    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls podName:0a8a8c4d-b364-4613-a6a1-7767c76a3efd nodeName:}" failed. No retries permitted until 2026-07-02 14:17:09.856665741 +0000 UTC m=+36.620051039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls") pod "dns-default-kbvnk" (UID: "0a8a8c4d-b364-4613-a6a1-7767c76a3efd") : secret "dns-default-metrics-tls" not found
Jul 02 14:17:08.970927 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:08.970890    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s2zwr" event={"ID":"a92336ee-ed07-4194-9236-03c1c360edbd","Type":"ContainerStarted","Data":"82e58668ec6df3b83980f5a0c1a9cbc37136a909936a02328bcdf74eadd5e226"}
Jul 02 14:17:09.744226 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.743853    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:17:09.744226 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.743864    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:09.744226 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.743992    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:09.746838 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.746814    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 14:17:09.746970 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.746885    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 14:17:09.748307 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.748152    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 14:17:09.748307 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.748196    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q6bk5\""
Jul 02 14:17:09.748307 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.748237    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 14:17:09.748307 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.748248    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8mqz5\""
Jul 02 14:17:09.863974 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:09.863941    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:09.864154 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:09.864101    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:17:09.864154 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:09.864154    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls podName:0a8a8c4d-b364-4613-a6a1-7767c76a3efd nodeName:}" failed. No retries permitted until 2026-07-02 14:17:11.864141076 +0000 UTC m=+38.627526377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls") pod "dns-default-kbvnk" (UID: "0a8a8c4d-b364-4613-a6a1-7767c76a3efd") : secret "dns-default-metrics-tls" not found
Jul 02 14:17:11.882546 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:11.882309    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:11.882957 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:11.882483    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:17:11.882957 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:11.882712    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls podName:0a8a8c4d-b364-4613-a6a1-7767c76a3efd nodeName:}" failed. No retries permitted until 2026-07-02 14:17:15.882689132 +0000 UTC m=+42.646074433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls") pod "dns-default-kbvnk" (UID: "0a8a8c4d-b364-4613-a6a1-7767c76a3efd") : secret "dns-default-metrics-tls" not found
Jul 02 14:17:12.982364 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:12.982334    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s2zwr" event={"ID":"a92336ee-ed07-4194-9236-03c1c360edbd","Type":"ContainerStarted","Data":"fa2051b16e368a0515316c4743b0e81d7ba60cbc593e57f43f9323cc7b69b685"}
Jul 02 14:17:12.985535 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:12.985509    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerStarted","Data":"03d07e9c4b15a9da55e99d47d446bef9de1b83ce7fba3b1d0ea59000d56fe9b1"}
Jul 02 14:17:13.001889 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:13.001845    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s2zwr" podStartSLOduration=0.844481483 podStartE2EDuration="5.001828032s" totalImagesPullingTime="4.157346549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:13.001507359 +0000 UTC m=+39.764892678" watchObservedRunningTime="2026-07-02 14:17:13.001828032 +0000 UTC m=+39.765213343"
Jul 02 14:17:13.990032 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:13.989994    2551 generic.go:356] "Generic (PLEG): container finished" podID="fae8e80b-baa0-4195-9657-aefc73c89147" containerID="03d07e9c4b15a9da55e99d47d446bef9de1b83ce7fba3b1d0ea59000d56fe9b1" exitCode=0
Jul 02 14:17:13.990379 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:13.990072    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerDied","Data":"03d07e9c4b15a9da55e99d47d446bef9de1b83ce7fba3b1d0ea59000d56fe9b1"}
Jul 02 14:17:14.995356 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:14.995324    2551 generic.go:356] "Generic (PLEG): container finished" podID="fae8e80b-baa0-4195-9657-aefc73c89147" containerID="7571f2be9596fadb1a183fb5e91685108123d0203394889886440516279daaf3" exitCode=0
Jul 02 14:17:14.995751 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:14.995398    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerDied","Data":"7571f2be9596fadb1a183fb5e91685108123d0203394889886440516279daaf3"}
Jul 02 14:17:15.912788 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:15.912756    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:15.912943 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:15.912893    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:17:15.912982 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:15.912946    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls podName:0a8a8c4d-b364-4613-a6a1-7767c76a3efd nodeName:}" failed. No retries permitted until 2026-07-02 14:17:23.912933343 +0000 UTC m=+50.676318641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls") pod "dns-default-kbvnk" (UID: "0a8a8c4d-b364-4613-a6a1-7767c76a3efd") : secret "dns-default-metrics-tls" not found
Jul 02 14:17:16.000842 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:16.000815    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" event={"ID":"fae8e80b-baa0-4195-9657-aefc73c89147","Type":"ContainerStarted","Data":"f23112e63b2d23771ce4d3fedf5438df0f50ffa03702e503f00034965913ceee"}
Jul 02 14:17:16.036606 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:16.036539    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jn8cz" podStartSLOduration=17.883936723 podStartE2EDuration="42.036516266s" totalImagesPullingTime="24.152579543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:16.034396254 +0000 UTC m=+42.797781572" watchObservedRunningTime="2026-07-02 14:17:16.036516266 +0000 UTC m=+42.799901570"
Jul 02 14:17:23.961033 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:23.961005    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:23.961532 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:23.961098    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:17:23.961532 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:23.961156    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls podName:0a8a8c4d-b364-4613-a6a1-7767c76a3efd nodeName:}" failed. No retries permitted until 2026-07-02 14:17:39.961143023 +0000 UTC m=+66.724528321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls") pod "dns-default-kbvnk" (UID: "0a8a8c4d-b364-4613-a6a1-7767c76a3efd") : secret "dns-default-metrics-tls" not found
Jul 02 14:17:35.971835 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:35.971809    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6krdm"
Jul 02 14:17:37.046778 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.046748    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"]
Jul 02 14:17:37.049056 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.049038    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.051877 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.051852    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 14:17:37.053418 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.053400    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 02 14:17:37.054547 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.054523    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 14:17:37.054782 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.054764    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-b62w4\""
Jul 02 14:17:37.054859 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.054815    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 02 14:17:37.061088 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.061068    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"]
Jul 02 14:17:37.146665 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.146632    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0fbe7466-cf96-4e32-be65-042c270a0f6f-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.146665 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.146666    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.146852 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.146685    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9mg\" (UniqueName: \"kubernetes.io/projected/0fbe7466-cf96-4e32-be65-042c270a0f6f-kube-api-access-zn9mg\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.247263 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.247228    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0fbe7466-cf96-4e32-be65-042c270a0f6f-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.247263 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.247268    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.247500 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.247292    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9mg\" (UniqueName: \"kubernetes.io/projected/0fbe7466-cf96-4e32-be65-042c270a0f6f-kube-api-access-zn9mg\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.247500 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:37.247396    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:37.247500 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:37.247489    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls podName:0fbe7466-cf96-4e32-be65-042c270a0f6f nodeName:}" failed. No retries permitted until 2026-07-02 14:17:37.747457384 +0000 UTC m=+64.510842685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-cmg89" (UID: "0fbe7466-cf96-4e32-be65-042c270a0f6f") : secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:37.248005 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.247985    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0fbe7466-cf96-4e32-be65-042c270a0f6f-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.259460 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.259437    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9mg\" (UniqueName: \"kubernetes.io/projected/0fbe7466-cf96-4e32-be65-042c270a0f6f-kube-api-access-zn9mg\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.751327 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:37.751296    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:37.751515 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:37.751389    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:37.751515 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:37.751448    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls podName:0fbe7466-cf96-4e32-be65-042c270a0f6f nodeName:}" failed. No retries permitted until 2026-07-02 14:17:38.751435123 +0000 UTC m=+65.514820422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-cmg89" (UID: "0fbe7466-cf96-4e32-be65-042c270a0f6f") : secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:38.758030 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:38.758003    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:38.758425 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:38.758147    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:38.758425 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:38.758216    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls podName:0fbe7466-cf96-4e32-be65-042c270a0f6f nodeName:}" failed. No retries permitted until 2026-07-02 14:17:40.758200631 +0000 UTC m=+67.521585933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-cmg89" (UID: "0fbe7466-cf96-4e32-be65-042c270a0f6f") : secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:39.564009 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.563968    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:17:39.564199 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.564029    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:39.564199 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.564064    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:39.566655 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.566627    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 14:17:39.566655 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.566650    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 14:17:39.566872 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.566860    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 14:17:39.574995 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:39.574974    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 02 14:17:39.575106 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:39.575027    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs podName:79b5c08b-176c-4374-a72b-97d488dcc046 nodeName:}" failed. No retries permitted until 2026-07-02 14:18:43.575010877 +0000 UTC m=+130.338396176 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs") pod "network-metrics-daemon-gr9j2" (UID: "79b5c08b-176c-4374-a72b-97d488dcc046") : secret "metrics-daemon-secret" not found
Jul 02 14:17:39.577095 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.577075    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3e26069-93f9-43ce-9605-f1da44d37f09-original-pull-secret\") pod \"global-pull-secret-syncer-qzshm\" (UID: \"e3e26069-93f9-43ce-9605-f1da44d37f09\") " pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:39.577212 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.577195    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 14:17:39.587542 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.587522    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65r9n\" (UniqueName: \"kubernetes.io/projected/065c8aea-2847-43cd-bff7-ba1e706a032d-kube-api-access-65r9n\") pod \"network-check-target-jpqgc\" (UID: \"065c8aea-2847-43cd-bff7-ba1e706a032d\") " pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:39.766355 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.766329    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzshm"
Jul 02 14:17:39.775566 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.775539    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q6bk5\""
Jul 02 14:17:39.782838 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.782819    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:39.894070 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.894031    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qzshm"]
Jul 02 14:17:39.897119 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:17:39.897094    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e26069_93f9_43ce_9605_f1da44d37f09.slice/crio-25250be181cf9610a7574c60f7a09b6b94ecb5380c8babdec3bf64075124b740 WatchSource:0}: Error finding container 25250be181cf9610a7574c60f7a09b6b94ecb5380c8babdec3bf64075124b740: Status 404 returned error can't find the container with id 25250be181cf9610a7574c60f7a09b6b94ecb5380c8babdec3bf64075124b740
Jul 02 14:17:39.912579 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.912557    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jpqgc"]
Jul 02 14:17:39.915198 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:17:39.915179    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065c8aea_2847_43cd_bff7_ba1e706a032d.slice/crio-97348478d1be52840dac889a3cfb2612e49297aa6db2494a1a5c54efd00c7d41 WatchSource:0}: Error finding container 97348478d1be52840dac889a3cfb2612e49297aa6db2494a1a5c54efd00c7d41: Status 404 returned error can't find the container with id 97348478d1be52840dac889a3cfb2612e49297aa6db2494a1a5c54efd00c7d41
Jul 02 14:17:39.966319 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:39.966287    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:17:39.966458 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:39.966440    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:17:39.966547 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:39.966523    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls podName:0a8a8c4d-b364-4613-a6a1-7767c76a3efd nodeName:}" failed. No retries permitted until 2026-07-02 14:18:11.966503067 +0000 UTC m=+98.729888369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls") pod "dns-default-kbvnk" (UID: "0a8a8c4d-b364-4613-a6a1-7767c76a3efd") : secret "dns-default-metrics-tls" not found
Jul 02 14:17:40.058432 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:40.058394    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jpqgc" event={"ID":"065c8aea-2847-43cd-bff7-ba1e706a032d","Type":"ContainerStarted","Data":"97348478d1be52840dac889a3cfb2612e49297aa6db2494a1a5c54efd00c7d41"}
Jul 02 14:17:40.059504 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:40.059481    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qzshm" event={"ID":"e3e26069-93f9-43ce-9605-f1da44d37f09","Type":"ContainerStarted","Data":"25250be181cf9610a7574c60f7a09b6b94ecb5380c8babdec3bf64075124b740"}
Jul 02 14:17:40.772033 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:40.771998    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:40.772526 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:40.772197    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:40.772526 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:40.772277    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls podName:0fbe7466-cf96-4e32-be65-042c270a0f6f nodeName:}" failed. No retries permitted until 2026-07-02 14:17:44.772255754 +0000 UTC m=+71.535641055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-cmg89" (UID: "0fbe7466-cf96-4e32-be65-042c270a0f6f") : secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:41.020915 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.020875    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"]
Jul 02 14:17:41.022905 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.022837    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.025462 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.025439    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 02 14:17:41.026898 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.026876    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nz2kp\""
Jul 02 14:17:41.026898 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.026897    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 02 14:17:41.027074 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.027040    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 14:17:41.027166 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.027151    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 02 14:17:41.032157 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.031673    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"]
Jul 02 14:17:41.074917 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.074881    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrx7\" (UniqueName: \"kubernetes.io/projected/9a7310e0-57f5-4205-80ff-b2c80168b89a-kube-api-access-xwrx7\") pod \"service-ca-operator-5d7b7dc95c-f7x7t\" (UID: \"9a7310e0-57f5-4205-80ff-b2c80168b89a\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.075083 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.074951    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a7310e0-57f5-4205-80ff-b2c80168b89a-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-f7x7t\" (UID: \"9a7310e0-57f5-4205-80ff-b2c80168b89a\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.075083 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.075014    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a7310e0-57f5-4205-80ff-b2c80168b89a-config\") pod \"service-ca-operator-5d7b7dc95c-f7x7t\" (UID: \"9a7310e0-57f5-4205-80ff-b2c80168b89a\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.175504 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.175460    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrx7\" (UniqueName: \"kubernetes.io/projected/9a7310e0-57f5-4205-80ff-b2c80168b89a-kube-api-access-xwrx7\") pod \"service-ca-operator-5d7b7dc95c-f7x7t\" (UID: \"9a7310e0-57f5-4205-80ff-b2c80168b89a\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.175685 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.175539    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a7310e0-57f5-4205-80ff-b2c80168b89a-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-f7x7t\" (UID: \"9a7310e0-57f5-4205-80ff-b2c80168b89a\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.175685 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.175584    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a7310e0-57f5-4205-80ff-b2c80168b89a-config\") pod \"service-ca-operator-5d7b7dc95c-f7x7t\" (UID: \"9a7310e0-57f5-4205-80ff-b2c80168b89a\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.176081 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.176053    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a7310e0-57f5-4205-80ff-b2c80168b89a-config\") pod \"service-ca-operator-5d7b7dc95c-f7x7t\" (UID: \"9a7310e0-57f5-4205-80ff-b2c80168b89a\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.178633 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.178583    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a7310e0-57f5-4205-80ff-b2c80168b89a-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-f7x7t\" (UID: \"9a7310e0-57f5-4205-80ff-b2c80168b89a\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.186254 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.186227    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrx7\" (UniqueName: \"kubernetes.io/projected/9a7310e0-57f5-4205-80ff-b2c80168b89a-kube-api-access-xwrx7\") pod \"service-ca-operator-5d7b7dc95c-f7x7t\" (UID: \"9a7310e0-57f5-4205-80ff-b2c80168b89a\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.234986 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.234954    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-dkj7s"]
Jul 02 14:17:41.236951 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.236927    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-dkj7s"
Jul 02 14:17:41.239719 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.239698    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-44crl\""
Jul 02 14:17:41.245676 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.245655    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-dkj7s"]
Jul 02 14:17:41.276333 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.276238    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2252l\" (UniqueName: \"kubernetes.io/projected/382bbd1e-e1ed-4dc4-8e37-cdcf1df7d16c-kube-api-access-2252l\") pod \"network-check-source-56499946bc-dkj7s\" (UID: \"382bbd1e-e1ed-4dc4-8e37-cdcf1df7d16c\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-dkj7s"
Jul 02 14:17:41.335246 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.335139    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"
Jul 02 14:17:41.378167 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.377786    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2252l\" (UniqueName: \"kubernetes.io/projected/382bbd1e-e1ed-4dc4-8e37-cdcf1df7d16c-kube-api-access-2252l\") pod \"network-check-source-56499946bc-dkj7s\" (UID: \"382bbd1e-e1ed-4dc4-8e37-cdcf1df7d16c\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-dkj7s"
Jul 02 14:17:41.389663 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.389579    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2252l\" (UniqueName: \"kubernetes.io/projected/382bbd1e-e1ed-4dc4-8e37-cdcf1df7d16c-kube-api-access-2252l\") pod \"network-check-source-56499946bc-dkj7s\" (UID: \"382bbd1e-e1ed-4dc4-8e37-cdcf1df7d16c\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-dkj7s"
Jul 02 14:17:41.482899 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.482827    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t"]
Jul 02 14:17:41.486460 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:17:41.486427    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7310e0_57f5_4205_80ff_b2c80168b89a.slice/crio-964313d30e933a046c7e2e565de4eda4dce622e579be8cec33d08f60464059f0 WatchSource:0}: Error finding container 964313d30e933a046c7e2e565de4eda4dce622e579be8cec33d08f60464059f0: Status 404 returned error can't find the container with id 964313d30e933a046c7e2e565de4eda4dce622e579be8cec33d08f60464059f0
Jul 02 14:17:41.549820 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.549704    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-dkj7s"
Jul 02 14:17:41.695752 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.695654    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-dkj7s"]
Jul 02 14:17:41.699181 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:17:41.699144    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382bbd1e_e1ed_4dc4_8e37_cdcf1df7d16c.slice/crio-ceb4183a71951b3741a0f795671d0ff576370216d8e9c598a8ce777daa4a0f3b WatchSource:0}: Error finding container ceb4183a71951b3741a0f795671d0ff576370216d8e9c598a8ce777daa4a0f3b: Status 404 returned error can't find the container with id ceb4183a71951b3741a0f795671d0ff576370216d8e9c598a8ce777daa4a0f3b
Jul 02 14:17:41.958579 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.958492    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sh7pn"]
Jul 02 14:17:41.961180 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.961097    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:41.963525 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.963502    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 14:17:41.963644 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.963553    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nfp8j\""
Jul 02 14:17:41.963780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.963764    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 14:17:41.963871 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.963851    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 14:17:41.964663 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.964644    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 14:17:41.972994 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.972975    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sh7pn"]
Jul 02 14:17:41.983505 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.983440    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:41.983609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.983513    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:41.983668 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.983614    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nc48\" (UniqueName: \"kubernetes.io/projected/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-api-access-4nc48\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:41.983719 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.983705    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-crio-socket\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:41.983767 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:41.983749    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-data-volume\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.067374 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.067334    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-dkj7s" event={"ID":"382bbd1e-e1ed-4dc4-8e37-cdcf1df7d16c","Type":"ContainerStarted","Data":"ceb4183a71951b3741a0f795671d0ff576370216d8e9c598a8ce777daa4a0f3b"}
Jul 02 14:17:42.068673 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.068634    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t" event={"ID":"9a7310e0-57f5-4205-80ff-b2c80168b89a","Type":"ContainerStarted","Data":"964313d30e933a046c7e2e565de4eda4dce622e579be8cec33d08f60464059f0"}
Jul 02 14:17:42.085080 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.085053    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.085235 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.085092    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.085235 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.085140    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nc48\" (UniqueName: \"kubernetes.io/projected/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-api-access-4nc48\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.085235 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:42.085198    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 14:17:42.085389 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.085255    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-crio-socket\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.085389 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:42.085280    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:42.585258241 +0000 UTC m=+69.348643543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : configmap "kube-rbac-proxy" not found
Jul 02 14:17:42.085389 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.085204    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-crio-socket\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.085389 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.085332    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-data-volume\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.085389 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:42.085335    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:42.085666 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:42.085431    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:42.585415568 +0000 UTC m=+69.348800867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:42.085666 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.085641    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-data-volume\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.096353 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.096328    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nc48\" (UniqueName: \"kubernetes.io/projected/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-api-access-4nc48\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.287487 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.287427    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr"]
Jul 02 14:17:42.290667 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.290636    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr"
Jul 02 14:17:42.295333 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.295303    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-s95c2\""
Jul 02 14:17:42.295899 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.295873    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 02 14:17:42.296286 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.296125    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 02 14:17:42.306231 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.306199    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr"]
Jul 02 14:17:42.388275 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.388170    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkz9\" (UniqueName: \"kubernetes.io/projected/3d582354-6c0a-4ac1-a13e-32ab9f7f48df-kube-api-access-fwkz9\") pod \"migrator-786f9fbc5-hcvfr\" (UID: \"3d582354-6c0a-4ac1-a13e-32ab9f7f48df\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr"
Jul 02 14:17:42.489292 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.489258    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkz9\" (UniqueName: \"kubernetes.io/projected/3d582354-6c0a-4ac1-a13e-32ab9f7f48df-kube-api-access-fwkz9\") pod \"migrator-786f9fbc5-hcvfr\" (UID: \"3d582354-6c0a-4ac1-a13e-32ab9f7f48df\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr"
Jul 02 14:17:42.501979 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.501948    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkz9\" (UniqueName: \"kubernetes.io/projected/3d582354-6c0a-4ac1-a13e-32ab9f7f48df-kube-api-access-fwkz9\") pod \"migrator-786f9fbc5-hcvfr\" (UID: \"3d582354-6c0a-4ac1-a13e-32ab9f7f48df\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr"
Jul 02 14:17:42.590428 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.590353    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.590428 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.590392    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:42.590636 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:42.590508    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 14:17:42.590636 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:42.590530    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:42.590636 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:42.590587    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:43.590565727 +0000 UTC m=+70.353951044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : configmap "kube-rbac-proxy" not found
Jul 02 14:17:42.590636 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:42.590617    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:43.590599188 +0000 UTC m=+70.353984490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:42.602322 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:42.602295    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr"
Jul 02 14:17:43.596999 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:43.596963    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:43.597452 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:43.597011    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:43.597452 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:43.597103    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 14:17:43.597452 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:43.597138    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:43.597452 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:43.597195    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:45.597173563 +0000 UTC m=+72.360558862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : configmap "kube-rbac-proxy" not found
Jul 02 14:17:43.597452 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:43.597216    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:45.597206449 +0000 UTC m=+72.360591754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:44.124784 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:44.124752    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z2vnx_9c7b1df8-a5d3-4087-9362-725f43f825d6/dns-node-resolver/0.log"
Jul 02 14:17:44.724562 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:44.724535    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4f9jp_bf27d440-b641-4efc-9a92-273c9bbef679/node-ca/0.log"
Jul 02 14:17:44.806655 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:44.806629    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:44.806827 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:44.806808    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:44.806903 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:44.806891    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls podName:0fbe7466-cf96-4e32-be65-042c270a0f6f nodeName:}" failed. No retries permitted until 2026-07-02 14:17:52.806869269 +0000 UTC m=+79.570254571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-cmg89" (UID: "0fbe7466-cf96-4e32-be65-042c270a0f6f") : secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:45.010422 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:45.010313    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr"]
Jul 02 14:17:45.022834 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:17:45.022804    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d582354_6c0a_4ac1_a13e_32ab9f7f48df.slice/crio-ad48fcd15dd650341de2d9383f512af5af919c5dcccdf9da7681eca49a0e40bc WatchSource:0}: Error finding container ad48fcd15dd650341de2d9383f512af5af919c5dcccdf9da7681eca49a0e40bc: Status 404 returned error can't find the container with id ad48fcd15dd650341de2d9383f512af5af919c5dcccdf9da7681eca49a0e40bc
Jul 02 14:17:45.080526 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:45.080491    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jpqgc" event={"ID":"065c8aea-2847-43cd-bff7-ba1e706a032d","Type":"ContainerStarted","Data":"c1c233974b7679ecb32496424939771ee8f93b3e7928f542f611b5bce65bafb6"}
Jul 02 14:17:45.080752 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:45.080733    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:17:45.082857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:45.082796    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr" event={"ID":"3d582354-6c0a-4ac1-a13e-32ab9f7f48df","Type":"ContainerStarted","Data":"ad48fcd15dd650341de2d9383f512af5af919c5dcccdf9da7681eca49a0e40bc"}
Jul 02 14:17:45.099172 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:45.099130    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jpqgc" podStartSLOduration=66.077378039 podStartE2EDuration="1m11.099114803s" totalImagesPullingTime="5.021736764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:45.097708267 +0000 UTC m=+71.861093590" watchObservedRunningTime="2026-07-02 14:17:45.099114803 +0000 UTC m=+71.862500120"
Jul 02 14:17:45.614614 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:45.614577    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:45.614803 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:45.614621    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:45.614803 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:45.614737    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:45.614803 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:45.614802    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:49.614782404 +0000 UTC m=+76.378167723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:45.614974 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:45.614809    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 14:17:45.614974 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:45.614847    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:49.614836497 +0000 UTC m=+76.378221795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : configmap "kube-rbac-proxy" not found
Jul 02 14:17:45.925777 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:45.925692    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s2zwr_a92336ee-ed07-4194-9236-03c1c360edbd/serve-healthcheck-canary/0.log"
Jul 02 14:17:46.091252 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:46.091213    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qzshm" event={"ID":"e3e26069-93f9-43ce-9605-f1da44d37f09","Type":"ContainerStarted","Data":"951018b0ac5e709d685e06e7d18d816722d9793c7e33adce8b94bd52050f1b68"}
Jul 02 14:17:46.093776 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:46.093715    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-dkj7s" event={"ID":"382bbd1e-e1ed-4dc4-8e37-cdcf1df7d16c","Type":"ContainerStarted","Data":"1e3f1ba4930d3397fa339c3c9ad635dec3a4e5eb3a1c4571136a4685f6b32681"}
Jul 02 14:17:46.108985 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:46.108929    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qzshm" podStartSLOduration=67.068353446 podStartE2EDuration="1m12.108910375s" totalImagesPullingTime="5.040556929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:46.106852146 +0000 UTC m=+72.870237480" watchObservedRunningTime="2026-07-02 14:17:46.108910375 +0000 UTC m=+72.872295697"
Jul 02 14:17:46.123695 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:46.123642    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56499946bc-dkj7s" podStartSLOduration=1.884859129 podStartE2EDuration="5.123627673s" totalImagesPullingTime="3.238768544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:46.122482048 +0000 UTC m=+72.885867361" watchObservedRunningTime="2026-07-02 14:17:46.123627673 +0000 UTC m=+72.887012993"
Jul 02 14:17:47.098524 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:47.098488    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t" event={"ID":"9a7310e0-57f5-4205-80ff-b2c80168b89a","Type":"ContainerStarted","Data":"8f46a5d35a4dd5c119ddfb79930abd67f28d43554ea57f5fd036f82c44e413fb"}
Jul 02 14:17:47.100273 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:47.100250    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr" event={"ID":"3d582354-6c0a-4ac1-a13e-32ab9f7f48df","Type":"ContainerStarted","Data":"8a23115bd96a5c1f13baa479550192188604322a6e4ad3eba4314f9ca1155a69"}
Jul 02 14:17:47.100394 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:47.100294    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr" event={"ID":"3d582354-6c0a-4ac1-a13e-32ab9f7f48df","Type":"ContainerStarted","Data":"5cfce3683357bd35e4992ce7c1f86d679af7efbd455ca6b52867b99d7a2384e4"}
Jul 02 14:17:47.117950 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:47.117892    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t" podStartSLOduration=0.676769216 podStartE2EDuration="6.117877151s" totalImagesPullingTime="5.441107935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:47.116620835 +0000 UTC m=+73.880006156" watchObservedRunningTime="2026-07-02 14:17:47.117877151 +0000 UTC m=+73.881262472"
Jul 02 14:17:47.136972 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:47.136928    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hcvfr" podStartSLOduration=3.233282119 podStartE2EDuration="5.136913426s" totalImagesPullingTime="1.903631307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:47.135939347 +0000 UTC m=+73.899324658" watchObservedRunningTime="2026-07-02 14:17:47.136913426 +0000 UTC m=+73.900298773"
Jul 02 14:17:49.642738 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:49.642695    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:49.643142 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:49.642747    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:49.643142 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:49.642828    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 14:17:49.643142 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:49.642915    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:49.643142 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:49.642934    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:57.642917097 +0000 UTC m=+84.406302395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : configmap "kube-rbac-proxy" not found
Jul 02 14:17:49.643142 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:49.642976    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:17:57.642961887 +0000 UTC m=+84.406347185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:50.729798 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.729768    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"]
Jul 02 14:17:50.732829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.732813    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:50.735373 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.735351    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 02 14:17:50.735669 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.735650    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 02 14:17:50.736761 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.736738    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ldvrr\""
Jul 02 14:17:50.736869 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.736764    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 02 14:17:50.736869 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.736823    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 02 14:17:50.742650 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.742624    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"]
Jul 02 14:17:50.851194 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.851149    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqwz\" (UniqueName: \"kubernetes.io/projected/4f17bd95-949b-4469-a744-ccb3e122845a-kube-api-access-mvqwz\") pod \"service-ca-7bb4f4bcc7-czfbk\" (UID: \"4f17bd95-949b-4469-a744-ccb3e122845a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:50.851363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.851232    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4f17bd95-949b-4469-a744-ccb3e122845a-signing-key\") pod \"service-ca-7bb4f4bcc7-czfbk\" (UID: \"4f17bd95-949b-4469-a744-ccb3e122845a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:50.851363 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.851300    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4f17bd95-949b-4469-a744-ccb3e122845a-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-czfbk\" (UID: \"4f17bd95-949b-4469-a744-ccb3e122845a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:50.952306 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.952275    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqwz\" (UniqueName: \"kubernetes.io/projected/4f17bd95-949b-4469-a744-ccb3e122845a-kube-api-access-mvqwz\") pod \"service-ca-7bb4f4bcc7-czfbk\" (UID: \"4f17bd95-949b-4469-a744-ccb3e122845a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:50.952465 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.952327    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4f17bd95-949b-4469-a744-ccb3e122845a-signing-key\") pod \"service-ca-7bb4f4bcc7-czfbk\" (UID: \"4f17bd95-949b-4469-a744-ccb3e122845a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:50.952465 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.952388    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4f17bd95-949b-4469-a744-ccb3e122845a-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-czfbk\" (UID: \"4f17bd95-949b-4469-a744-ccb3e122845a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:50.953037 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.953012    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4f17bd95-949b-4469-a744-ccb3e122845a-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-czfbk\" (UID: \"4f17bd95-949b-4469-a744-ccb3e122845a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:50.954680 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.954660    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4f17bd95-949b-4469-a744-ccb3e122845a-signing-key\") pod \"service-ca-7bb4f4bcc7-czfbk\" (UID: \"4f17bd95-949b-4469-a744-ccb3e122845a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:50.961502 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:50.961463    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqwz\" (UniqueName: \"kubernetes.io/projected/4f17bd95-949b-4469-a744-ccb3e122845a-kube-api-access-mvqwz\") pod \"service-ca-7bb4f4bcc7-czfbk\" (UID: \"4f17bd95-949b-4469-a744-ccb3e122845a\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:51.041603 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:51.041561    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"
Jul 02 14:17:51.162736 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:51.162705    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-czfbk"]
Jul 02 14:17:51.166627 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:17:51.166603    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f17bd95_949b_4469_a744_ccb3e122845a.slice/crio-793dcf924406536cc522ce3dc11f01b669577e1ec020be0efe9103c4ce9aba24 WatchSource:0}: Error finding container 793dcf924406536cc522ce3dc11f01b669577e1ec020be0efe9103c4ce9aba24: Status 404 returned error can't find the container with id 793dcf924406536cc522ce3dc11f01b669577e1ec020be0efe9103c4ce9aba24
Jul 02 14:17:52.118099 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:52.118060    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk" event={"ID":"4f17bd95-949b-4469-a744-ccb3e122845a","Type":"ContainerStarted","Data":"fde264abdd882cbd4ba221caa17fe4f2bfecb1ef2359dfeaca8e40505cebc2cf"}
Jul 02 14:17:52.118556 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:52.118112    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk" event={"ID":"4f17bd95-949b-4469-a744-ccb3e122845a","Type":"ContainerStarted","Data":"793dcf924406536cc522ce3dc11f01b669577e1ec020be0efe9103c4ce9aba24"}
Jul 02 14:17:52.144605 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:52.144557    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-czfbk" podStartSLOduration=2.144541036 podStartE2EDuration="2.144541036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:52.144226176 +0000 UTC m=+78.907611497" watchObservedRunningTime="2026-07-02 14:17:52.144541036 +0000 UTC m=+78.907926357"
Jul 02 14:17:52.868052 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:52.868018    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:17:52.868319 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:52.868179    2551 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:52.868464 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:52.868454    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls podName:0fbe7466-cf96-4e32-be65-042c270a0f6f nodeName:}" failed. No retries permitted until 2026-07-02 14:18:08.8684334 +0000 UTC m=+95.631818706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-cmg89" (UID: "0fbe7466-cf96-4e32-be65-042c270a0f6f") : secret "cluster-monitoring-operator-tls" not found
Jul 02 14:17:57.708523 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:57.708397    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:57.708523 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:57.708441    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:17:57.708935 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:57.708607    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:57.708935 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:17:57.708685    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls podName:f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3 nodeName:}" failed. No retries permitted until 2026-07-02 14:18:13.70866719 +0000 UTC m=+100.472052505 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sh7pn" (UID: "f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:17:57.709011 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:17:57.708937    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:18:08.895681 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:08.895629    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:18:08.898003 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:08.897978    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0fbe7466-cf96-4e32-be65-042c270a0f6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-cmg89\" (UID: \"0fbe7466-cf96-4e32-be65-042c270a0f6f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:18:09.157282 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:09.157202    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"
Jul 02 14:18:09.279426 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:09.279270    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89"]
Jul 02 14:18:09.281799 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:18:09.281772    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbe7466_cf96_4e32_be65_042c270a0f6f.slice/crio-42d965b2ea8c8360126f6e34b7ddf2d956615b5e99dff62d0e11b770cec6741a WatchSource:0}: Error finding container 42d965b2ea8c8360126f6e34b7ddf2d956615b5e99dff62d0e11b770cec6741a: Status 404 returned error can't find the container with id 42d965b2ea8c8360126f6e34b7ddf2d956615b5e99dff62d0e11b770cec6741a
Jul 02 14:18:10.174201 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:10.174166    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89" event={"ID":"0fbe7466-cf96-4e32-be65-042c270a0f6f","Type":"ContainerStarted","Data":"42d965b2ea8c8360126f6e34b7ddf2d956615b5e99dff62d0e11b770cec6741a"}
Jul 02 14:18:11.684543 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.684505    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"]
Jul 02 14:18:11.687743 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.687726    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.690537 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.690513    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 14:18:11.690537 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.690529    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 14:18:11.690712 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.690634    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 02 14:18:11.690712 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.690662    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 02 14:18:11.690820 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.690792    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 02 14:18:11.691853 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.691840    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 14:18:11.691904 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.691860    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 02 14:18:11.700510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.700481    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"]
Jul 02 14:18:11.817428 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.817389    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-ca\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.817428 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.817429    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.817677 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.817453    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.817677 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.817492    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rttxd\" (UniqueName: \"kubernetes.io/projected/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-kube-api-access-rttxd\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.817677 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.817549    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-hub\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.817677 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.817588    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.918369 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.918193    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-ca\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.918369 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.918245    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.918707 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.918682    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.918833 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.918726    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rttxd\" (UniqueName: \"kubernetes.io/projected/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-kube-api-access-rttxd\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.918833 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.918777    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-hub\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.918939 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.918833    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.919028 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.919009    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.921773 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.921596    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-ca\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.921884 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.921809    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.922487 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.922439    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.922584 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.922523    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-hub\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:11.933856 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:11.933822    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rttxd\" (UniqueName: \"kubernetes.io/projected/4193cc96-21e3-4c7b-84c4-25cea4f0bd28-kube-api-access-rttxd\") pod \"cluster-proxy-proxy-agent-64c4dcdb78-q7n6z\" (UID: \"4193cc96-21e3-4c7b-84c4-25cea4f0bd28\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:12.007957 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:12.007929    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"
Jul 02 14:18:12.019350 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:12.019315    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:18:12.022010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:12.021985    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a8a8c4d-b364-4613-a6a1-7767c76a3efd-metrics-tls\") pod \"dns-default-kbvnk\" (UID: \"0a8a8c4d-b364-4613-a6a1-7767c76a3efd\") " pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:18:12.040336 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:12.040305    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mttc5\""
Jul 02 14:18:12.047973 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:12.047944    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:18:12.184174 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:12.184144    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89" event={"ID":"0fbe7466-cf96-4e32-be65-042c270a0f6f","Type":"ContainerStarted","Data":"28e108132384b18b3973f35476053ddbb5de746cf9285c45813816f5314279ce"}
Jul 02 14:18:12.194181 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:12.194156    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z"]
Jul 02 14:18:12.211212 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:12.211165    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-cmg89" podStartSLOduration=33.252840309 podStartE2EDuration="35.211148343s" totalImagesPullingTime="1.958308034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:18:12.209893288 +0000 UTC m=+98.973278611" watchObservedRunningTime="2026-07-02 14:18:12.211148343 +0000 UTC m=+98.974533664"
Jul 02 14:18:12.225243 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:12.225216    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kbvnk"]
Jul 02 14:18:12.228279 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:18:12.228254    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8a8c4d_b364_4613_a6a1_7767c76a3efd.slice/crio-5688f14e642f0a75b0442e34018ab35fe479a878a00fe759962a7dc9ad61b635 WatchSource:0}: Error finding container 5688f14e642f0a75b0442e34018ab35fe479a878a00fe759962a7dc9ad61b635: Status 404 returned error can't find the container with id 5688f14e642f0a75b0442e34018ab35fe479a878a00fe759962a7dc9ad61b635
Jul 02 14:18:13.189947 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:13.189905    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z" event={"ID":"4193cc96-21e3-4c7b-84c4-25cea4f0bd28","Type":"ContainerStarted","Data":"0193232b837dbea053df739b56af74c390dd2b38c3f64c74db00e1f2127371c5"}
Jul 02 14:18:13.191287 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:13.191258    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbvnk" event={"ID":"0a8a8c4d-b364-4613-a6a1-7767c76a3efd","Type":"ContainerStarted","Data":"5688f14e642f0a75b0442e34018ab35fe479a878a00fe759962a7dc9ad61b635"}
Jul 02 14:18:13.735388 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:13.735349    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:18:13.741732 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:13.740714    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sh7pn\" (UID: \"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3\") " pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:18:13.773385 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:13.773352    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sh7pn"
Jul 02 14:18:13.938269 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:13.938234    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sh7pn"]
Jul 02 14:18:14.726059 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:18:14.726013    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90b6ffa_cfd9_44bf_bf17_7b9da1c50ff3.slice/crio-394fc269cbd3717557649cbb16a6fce906c828a7edfa5fa937ff6cc867ec7215 WatchSource:0}: Error finding container 394fc269cbd3717557649cbb16a6fce906c828a7edfa5fa937ff6cc867ec7215: Status 404 returned error can't find the container with id 394fc269cbd3717557649cbb16a6fce906c828a7edfa5fa937ff6cc867ec7215
Jul 02 14:18:15.200377 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:15.200346    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh7pn" event={"ID":"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3","Type":"ContainerStarted","Data":"394fc269cbd3717557649cbb16a6fce906c828a7edfa5fa937ff6cc867ec7215"}
Jul 02 14:18:16.097198 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:16.097155    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jpqgc"
Jul 02 14:18:17.210767 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:17.210725    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z" event={"ID":"4193cc96-21e3-4c7b-84c4-25cea4f0bd28","Type":"ContainerStarted","Data":"586a7738c798deed86cb1ab003c320fd462e12c5745e1b3baee30fdb3fa1a2f5"}
Jul 02 14:18:17.211248 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:17.210780    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z" event={"ID":"4193cc96-21e3-4c7b-84c4-25cea4f0bd28","Type":"ContainerStarted","Data":"5994cf0096d1f8171dbbdc2e9b815ab326f74ede0401aba2d41427853b27557e"}
Jul 02 14:18:17.211248 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:17.210799    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z" event={"ID":"4193cc96-21e3-4c7b-84c4-25cea4f0bd28","Type":"ContainerStarted","Data":"f13621b255feed795cfb581c4092416e4475915a29404f7eb99876d7e33cff02"}
Jul 02 14:18:17.212656 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:17.212630    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbvnk" event={"ID":"0a8a8c4d-b364-4613-a6a1-7767c76a3efd","Type":"ContainerStarted","Data":"1c33dbba7d3b40bd47d14348f34316b396dd4642d1d66f6ad874ca5c88875bde"}
Jul 02 14:18:17.212778 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:17.212683    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kbvnk" event={"ID":"0a8a8c4d-b364-4613-a6a1-7767c76a3efd","Type":"ContainerStarted","Data":"1674d0cf031c166c3e092c3dfc73690888d3b3539c4937837f984969cd7c6d21"}
Jul 02 14:18:17.212778 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:17.212721    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:18:17.235608 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:17.235545    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64c4dcdb78-q7n6z" podStartSLOduration=2.06037999 podStartE2EDuration="6.235523706s" totalImagesPullingTime="4.175143716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:18:17.234506859 +0000 UTC m=+103.997892177" watchObservedRunningTime="2026-07-02 14:18:17.235523706 +0000 UTC m=+103.998909028"
Jul 02 14:18:17.257044 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:17.256994    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kbvnk" podStartSLOduration=65.125091139 podStartE2EDuration="1m9.256976587s" totalImagesPullingTime="4.131885448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:18:17.256144268 +0000 UTC m=+104.019529605" watchObservedRunningTime="2026-07-02 14:18:17.256976587 +0000 UTC m=+104.020361907"
Jul 02 14:18:21.228560 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:21.228519    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh7pn" event={"ID":"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3","Type":"ContainerStarted","Data":"076f5f3e14af435ac10dd9ac653318f98313dc5c3bed2a8699716814d221a611"}
Jul 02 14:18:22.235033 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.235005    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh7pn" event={"ID":"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3","Type":"ContainerStarted","Data":"9b0e4963fefd6e0ee56b70a9d118569ae89a7676864900722d05c08742aaaf50"}
Jul 02 14:18:22.448620 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.448535    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r9m2n"]
Jul 02 14:18:22.450596 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.450566    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.454660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.454637    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 14:18:22.455002 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.454984    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 14:18:22.455085 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.455041    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 14:18:22.456211 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.456193    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 14:18:22.456334 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.456245    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h5vwt\""
Jul 02 14:18:22.614673 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.614632    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-textfile\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.614873 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.614701    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-root\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.614873 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.614782    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.614873 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.614830    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trcrx\" (UniqueName: \"kubernetes.io/projected/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-kube-api-access-trcrx\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.614873 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.614861    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-wtmp\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.615071 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.614903    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-tls\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.615071 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.614927    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-metrics-client-ca\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.615071 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.614976    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.615071 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.615014    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-sys\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.716412 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716317    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.716412 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716369    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-sys\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716416    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-textfile\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716458    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-root\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716507    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716543    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trcrx\" (UniqueName: \"kubernetes.io/projected/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-kube-api-access-trcrx\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716552    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-sys\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716571    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-wtmp\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716617    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-tls\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716641    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-metrics-client-ca\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716651    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-root\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.716808    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-wtmp\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:18:22.716896    2551 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:18:22.716974    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-tls podName:06cdddc9-e1d5-4d17-8edc-bbc4d6936508 nodeName:}" failed. No retries permitted until 2026-07-02 14:18:23.216955324 +0000 UTC m=+109.980340623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-tls") pod "node-exporter-r9m2n" (UID: "06cdddc9-e1d5-4d17-8edc-bbc4d6936508") : secret "node-exporter-tls" not found
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.717247    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-textfile\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.717307    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.717510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.717499    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-metrics-client-ca\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.720704 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.720678    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:22.728078 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:22.727649    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trcrx\" (UniqueName: \"kubernetes.io/projected/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-kube-api-access-trcrx\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:23.222582 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:23.222546    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-tls\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:23.225349 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:23.225325    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06cdddc9-e1d5-4d17-8edc-bbc4d6936508-node-exporter-tls\") pod \"node-exporter-r9m2n\" (UID: \"06cdddc9-e1d5-4d17-8edc-bbc4d6936508\") " pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:23.362003 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:23.361957    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r9m2n"
Jul 02 14:18:23.374141 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:18:23.374098    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cdddc9_e1d5_4d17_8edc_bbc4d6936508.slice/crio-639544bb9498afdd3ea93d7af1ce4a750840c40a97a05e4e92042221a29a274d WatchSource:0}: Error finding container 639544bb9498afdd3ea93d7af1ce4a750840c40a97a05e4e92042221a29a274d: Status 404 returned error can't find the container with id 639544bb9498afdd3ea93d7af1ce4a750840c40a97a05e4e92042221a29a274d
Jul 02 14:18:24.242799 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:24.242754    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9m2n" event={"ID":"06cdddc9-e1d5-4d17-8edc-bbc4d6936508","Type":"ContainerStarted","Data":"639544bb9498afdd3ea93d7af1ce4a750840c40a97a05e4e92042221a29a274d"}
Jul 02 14:18:25.247789 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:25.247756    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sh7pn" event={"ID":"f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3","Type":"ContainerStarted","Data":"93c4f015afdb4d511650633377c402a7d7e878e087a504ad3efb47c1a911d92a"}
Jul 02 14:18:25.271969 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:25.271919    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sh7pn" podStartSLOduration=34.374723007 podStartE2EDuration="44.271903626s" totalImagesPullingTime="9.897180619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:17:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:18:25.270371518 +0000 UTC m=+112.033756840" watchObservedRunningTime="2026-07-02 14:18:25.271903626 +0000 UTC m=+112.035288970"
Jul 02 14:18:26.252809 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:26.252777    2551 generic.go:356] "Generic (PLEG): container finished" podID="06cdddc9-e1d5-4d17-8edc-bbc4d6936508" containerID="20902b1efc046ba70fc829b9ff2a0e7ef21a9880c0b871c0bb7cd3641437761c" exitCode=0
Jul 02 14:18:26.253178 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:26.252848    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9m2n" event={"ID":"06cdddc9-e1d5-4d17-8edc-bbc4d6936508","Type":"ContainerDied","Data":"20902b1efc046ba70fc829b9ff2a0e7ef21a9880c0b871c0bb7cd3641437761c"}
Jul 02 14:18:27.218236 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:27.218211    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kbvnk"
Jul 02 14:18:27.257679 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:27.257644    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9m2n" event={"ID":"06cdddc9-e1d5-4d17-8edc-bbc4d6936508","Type":"ContainerStarted","Data":"0728278c7ce13ba95b52fa27e10157c8f6c3855a88bd47acaaaaabc962ddaa84"}
Jul 02 14:18:27.258044 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:27.257696    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9m2n" event={"ID":"06cdddc9-e1d5-4d17-8edc-bbc4d6936508","Type":"ContainerStarted","Data":"0ad4569eaac813210d48c1c601f5f6f15f83e2fc0a0cd1baa74379e53bc6762b"}
Jul 02 14:18:27.286811 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:27.286757    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r9m2n" podStartSLOduration=3.315583104 podStartE2EDuration="5.28673801s" totalImagesPullingTime="1.899314208s" totalInitContainerRuntime="71.840698ms" isStatefulPod=true podCreationTimestamp="2026-07-02 14:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:18:27.28618659 +0000 UTC m=+114.049571905" watchObservedRunningTime="2026-07-02 14:18:27.28673801 +0000 UTC m=+114.050123330"
Jul 02 14:18:35.159205 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.159173    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-tnwc2"]
Jul 02 14:18:35.161704 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.161687    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-tnwc2"
Jul 02 14:18:35.164482 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.164447    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-25stn\""
Jul 02 14:18:35.164482 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.164456    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 14:18:35.164649 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.164445    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 14:18:35.170861 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.170831    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-tnwc2"]
Jul 02 14:18:35.206212 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.206173    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89277\" (UniqueName: \"kubernetes.io/projected/866aca15-9498-48dd-bee5-7504840402f1-kube-api-access-89277\") pod \"downloads-5499b6fb8f-tnwc2\" (UID: \"866aca15-9498-48dd-bee5-7504840402f1\") " pod="openshift-console/downloads-5499b6fb8f-tnwc2"
Jul 02 14:18:35.307425 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.307385    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89277\" (UniqueName: \"kubernetes.io/projected/866aca15-9498-48dd-bee5-7504840402f1-kube-api-access-89277\") pod \"downloads-5499b6fb8f-tnwc2\" (UID: \"866aca15-9498-48dd-bee5-7504840402f1\") " pod="openshift-console/downloads-5499b6fb8f-tnwc2"
Jul 02 14:18:35.318233 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.318211    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89277\" (UniqueName: \"kubernetes.io/projected/866aca15-9498-48dd-bee5-7504840402f1-kube-api-access-89277\") pod \"downloads-5499b6fb8f-tnwc2\" (UID: \"866aca15-9498-48dd-bee5-7504840402f1\") " pod="openshift-console/downloads-5499b6fb8f-tnwc2"
Jul 02 14:18:35.470216 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.470130    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-tnwc2"
Jul 02 14:18:35.602924 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:35.602898    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-tnwc2"]
Jul 02 14:18:35.605033 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:18:35.605002    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866aca15_9498_48dd_bee5_7504840402f1.slice/crio-86145469984b682a23f2696f2c14baa6085c6875df273b960d5e803bb9dd8d51 WatchSource:0}: Error finding container 86145469984b682a23f2696f2c14baa6085c6875df273b960d5e803bb9dd8d51: Status 404 returned error can't find the container with id 86145469984b682a23f2696f2c14baa6085c6875df273b960d5e803bb9dd8d51
Jul 02 14:18:36.290748 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:36.290714    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-tnwc2" event={"ID":"866aca15-9498-48dd-bee5-7504840402f1","Type":"ContainerStarted","Data":"86145469984b682a23f2696f2c14baa6085c6875df273b960d5e803bb9dd8d51"}
Jul 02 14:18:43.583764 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:43.583719    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:18:43.586299 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:43.586272    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b5c08b-176c-4374-a72b-97d488dcc046-metrics-certs\") pod \"network-metrics-daemon-gr9j2\" (UID: \"79b5c08b-176c-4374-a72b-97d488dcc046\") " pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:18:43.660250 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:43.660211    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8mqz5\""
Jul 02 14:18:43.668569 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:43.668536    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr9j2"
Jul 02 14:18:43.825887 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:43.825854    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gr9j2"]
Jul 02 14:18:43.829169 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:18:43.829139    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b5c08b_176c_4374_a72b_97d488dcc046.slice/crio-130deb5a27dec58e4a5221f2862add45e2823c4a44f7ee9d9bc41cc82abd9ab6 WatchSource:0}: Error finding container 130deb5a27dec58e4a5221f2862add45e2823c4a44f7ee9d9bc41cc82abd9ab6: Status 404 returned error can't find the container with id 130deb5a27dec58e4a5221f2862add45e2823c4a44f7ee9d9bc41cc82abd9ab6
Jul 02 14:18:44.323290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:44.323258    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gr9j2" event={"ID":"79b5c08b-176c-4374-a72b-97d488dcc046","Type":"ContainerStarted","Data":"130deb5a27dec58e4a5221f2862add45e2823c4a44f7ee9d9bc41cc82abd9ab6"}
Jul 02 14:18:46.333956 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:46.333891    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gr9j2" event={"ID":"79b5c08b-176c-4374-a72b-97d488dcc046","Type":"ContainerStarted","Data":"37f0e85f6dd2002589b6a8db07ab66e7acde44b7c7cca3846a1f9121c1252470"}
Jul 02 14:18:46.334416 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:46.333982    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gr9j2" event={"ID":"79b5c08b-176c-4374-a72b-97d488dcc046","Type":"ContainerStarted","Data":"9de05c39eb23f9bfa89ccbbe5b3100c322f8cc186a04e6df79936dc6ed76b739"}
Jul 02 14:18:46.357744 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:46.357682    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gr9j2" podStartSLOduration=130.559246994 podStartE2EDuration="2m12.357661901s" totalImagesPullingTime="1.798414907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:18:46.3572507 +0000 UTC m=+133.120636037" watchObservedRunningTime="2026-07-02 14:18:46.357661901 +0000 UTC m=+133.121047226"
Jul 02 14:18:54.365641 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:54.365602    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-tnwc2" event={"ID":"866aca15-9498-48dd-bee5-7504840402f1","Type":"ContainerStarted","Data":"47e79b436c9390ae8bfeeeea5878531acf7c23c513b452dbe8bcdaf37f322939"}
Jul 02 14:18:54.366004 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:54.365810    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-tnwc2"
Jul 02 14:18:54.367175 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:54.367150    2551 patch_prober.go:28] interesting pod/downloads-5499b6fb8f-tnwc2 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body=
Jul 02 14:18:54.367277 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:54.367196    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5499b6fb8f-tnwc2" podUID="866aca15-9498-48dd-bee5-7504840402f1" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused"
Jul 02 14:18:55.386917 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:55.386884    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-tnwc2"
Jul 02 14:18:55.407482 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:18:55.407423    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-tnwc2" podStartSLOduration=1.874990925 podStartE2EDuration="20.407399592s" totalImagesPullingTime="18.532408667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:18:54.384681457 +0000 UTC m=+141.148066787" watchObservedRunningTime="2026-07-02 14:18:55.407399592 +0000 UTC m=+142.170784915"
Jul 02 14:19:23.472370 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:19:23.472338    2551 generic.go:356] "Generic (PLEG): container finished" podID="9a7310e0-57f5-4205-80ff-b2c80168b89a" containerID="8f46a5d35a4dd5c119ddfb79930abd67f28d43554ea57f5fd036f82c44e413fb" exitCode=0
Jul 02 14:19:23.472802 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:19:23.472388    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t" event={"ID":"9a7310e0-57f5-4205-80ff-b2c80168b89a","Type":"ContainerDied","Data":"8f46a5d35a4dd5c119ddfb79930abd67f28d43554ea57f5fd036f82c44e413fb"}
Jul 02 14:19:23.472802 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:19:23.472687    2551 scope.go:117] "RemoveContainer" containerID="8f46a5d35a4dd5c119ddfb79930abd67f28d43554ea57f5fd036f82c44e413fb"
Jul 02 14:19:24.477824 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:19:24.477787    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-f7x7t" event={"ID":"9a7310e0-57f5-4205-80ff-b2c80168b89a","Type":"ContainerStarted","Data":"89a04127c07530acac778a3b0cdad6e493deb9727cd4c3f8bfccf4ddad1c2677"}
Jul 02 14:21:31.797826 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.797790    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"]
Jul 02 14:21:31.799752 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.799736    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:31.804079 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.804057    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 14:21:31.805330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.805295    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 14:21:31.805451 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.805350    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 02 14:21:31.805451 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.805388    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 14:21:31.805451 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.805348    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 02 14:21:31.805625 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.805497    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-8cfr9\""
Jul 02 14:21:31.816629 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.816605    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"]
Jul 02 14:21:31.939225 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.939179    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f9373e0b-3588-4345-949f-748a7f0886a8-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:31.939225 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.939234    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f9373e0b-3588-4345-949f-748a7f0886a8-certificates\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:31.939444 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:31.939277    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6vlv\" (UniqueName: \"kubernetes.io/projected/f9373e0b-3588-4345-949f-748a7f0886a8-kube-api-access-x6vlv\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:32.040548 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.040515    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6vlv\" (UniqueName: \"kubernetes.io/projected/f9373e0b-3588-4345-949f-748a7f0886a8-kube-api-access-x6vlv\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:32.040700 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.040585    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f9373e0b-3588-4345-949f-748a7f0886a8-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:32.040758 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.040740    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f9373e0b-3588-4345-949f-748a7f0886a8-certificates\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:32.040857 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:21:32.040845    2551 secret.go:281] references non-existent secret key: tls.crt
Jul 02 14:21:32.040896 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:21:32.040861    2551 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 02 14:21:32.040896 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:21:32.040881    2551 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s: references non-existent secret key: tls.crt
Jul 02 14:21:32.040957 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:21:32.040934    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9373e0b-3588-4345-949f-748a7f0886a8-certificates podName:f9373e0b-3588-4345-949f-748a7f0886a8 nodeName:}" failed. No retries permitted until 2026-07-02 14:21:32.540917532 +0000 UTC m=+299.304302844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f9373e0b-3588-4345-949f-748a7f0886a8-certificates") pod "keda-metrics-apiserver-559f75f947-rsq8s" (UID: "f9373e0b-3588-4345-949f-748a7f0886a8") : references non-existent secret key: tls.crt
Jul 02 14:21:32.040998 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.040967    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f9373e0b-3588-4345-949f-748a7f0886a8-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:32.050384 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.050326    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6vlv\" (UniqueName: \"kubernetes.io/projected/f9373e0b-3588-4345-949f-748a7f0886a8-kube-api-access-x6vlv\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:32.544321 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.544277    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f9373e0b-3588-4345-949f-748a7f0886a8-certificates\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:32.546822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.546800    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f9373e0b-3588-4345-949f-748a7f0886a8-certificates\") pod \"keda-metrics-apiserver-559f75f947-rsq8s\" (UID: \"f9373e0b-3588-4345-949f-748a7f0886a8\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:32.710301 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.710263    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:32.841859 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.841836    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"]
Jul 02 14:21:32.844195 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:21:32.844169    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9373e0b_3588_4345_949f_748a7f0886a8.slice/crio-daf6d0f02bdb5b86abc9a2abccc86c2fe866e80074e6ea1ccae5ce9d78ca3c6d WatchSource:0}: Error finding container daf6d0f02bdb5b86abc9a2abccc86c2fe866e80074e6ea1ccae5ce9d78ca3c6d: Status 404 returned error can't find the container with id daf6d0f02bdb5b86abc9a2abccc86c2fe866e80074e6ea1ccae5ce9d78ca3c6d
Jul 02 14:21:32.908825 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:32.908788    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s" event={"ID":"f9373e0b-3588-4345-949f-748a7f0886a8","Type":"ContainerStarted","Data":"daf6d0f02bdb5b86abc9a2abccc86c2fe866e80074e6ea1ccae5ce9d78ca3c6d"}
Jul 02 14:21:33.691586 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:33.691554    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:21:33.696348 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:33.696327    2551 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 14:21:33.696816 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:33.696799    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:21:37.930106 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:37.930058    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s" event={"ID":"f9373e0b-3588-4345-949f-748a7f0886a8","Type":"ContainerStarted","Data":"6906f3f3cb2c49a1c4fe6d6309d279b99484df5dcea4961652a10dc3b705ad17"}
Jul 02 14:21:37.930641 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:37.930195    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:21:37.950639 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:37.950584    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s" podStartSLOduration=2.085153283 podStartE2EDuration="6.950568336s" totalImagesPullingTime="4.865415053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:21:37.949724079 +0000 UTC m=+304.713109400" watchObservedRunningTime="2026-07-02 14:21:37.950568336 +0000 UTC m=+304.713953656"
Jul 02 14:21:48.939208 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:21:48.939182    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-rsq8s"
Jul 02 14:22:38.435126 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.435090    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-t9sh9"]
Jul 02 14:22:38.438457 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.438439    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:22:38.441276 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.441249    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 14:22:38.442364 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.442339    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 14:22:38.442504 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.442339    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-ssdvt\""
Jul 02 14:22:38.442504 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.442347    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 14:22:38.447588 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.447565    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-t9sh9"]
Jul 02 14:22:38.473112 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.473080    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-dfxpj"]
Jul 02 14:22:38.475992 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.475972    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:22:38.478664 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.478636    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tgwzl\""
Jul 02 14:22:38.478780 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.478640    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 14:22:38.486499 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.486460    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-dfxpj"]
Jul 02 14:22:38.524145 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.524119    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzzm\" (UniqueName: \"kubernetes.io/projected/37afa26a-df0c-4a24-a157-d884de7614c8-kube-api-access-9kzzm\") pod \"seaweedfs-64568bcd49-dfxpj\" (UID: \"37afa26a-df0c-4a24-a157-d884de7614c8\") " pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:22:38.524288 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.524167    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/37afa26a-df0c-4a24-a157-d884de7614c8-data\") pod \"seaweedfs-64568bcd49-dfxpj\" (UID: \"37afa26a-df0c-4a24-a157-d884de7614c8\") " pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:22:38.524288 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.524195    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcgdp\" (UniqueName: \"kubernetes.io/projected/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-kube-api-access-dcgdp\") pod \"kserve-controller-manager-7b9c77fff9-t9sh9\" (UID: \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\") " pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:22:38.524288 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.524231    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-cert\") pod \"kserve-controller-manager-7b9c77fff9-t9sh9\" (UID: \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\") " pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:22:38.625078 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.625039    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/37afa26a-df0c-4a24-a157-d884de7614c8-data\") pod \"seaweedfs-64568bcd49-dfxpj\" (UID: \"37afa26a-df0c-4a24-a157-d884de7614c8\") " pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:22:38.625255 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.625091    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcgdp\" (UniqueName: \"kubernetes.io/projected/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-kube-api-access-dcgdp\") pod \"kserve-controller-manager-7b9c77fff9-t9sh9\" (UID: \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\") " pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:22:38.625255 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.625138    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-cert\") pod \"kserve-controller-manager-7b9c77fff9-t9sh9\" (UID: \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\") " pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:22:38.625255 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.625176    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzzm\" (UniqueName: \"kubernetes.io/projected/37afa26a-df0c-4a24-a157-d884de7614c8-kube-api-access-9kzzm\") pod \"seaweedfs-64568bcd49-dfxpj\" (UID: \"37afa26a-df0c-4a24-a157-d884de7614c8\") " pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:22:38.625410 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.625392    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/37afa26a-df0c-4a24-a157-d884de7614c8-data\") pod \"seaweedfs-64568bcd49-dfxpj\" (UID: \"37afa26a-df0c-4a24-a157-d884de7614c8\") " pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:22:38.627580 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.627554    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-cert\") pod \"kserve-controller-manager-7b9c77fff9-t9sh9\" (UID: \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\") " pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:22:38.634808 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.634788    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzzm\" (UniqueName: \"kubernetes.io/projected/37afa26a-df0c-4a24-a157-d884de7614c8-kube-api-access-9kzzm\") pod \"seaweedfs-64568bcd49-dfxpj\" (UID: \"37afa26a-df0c-4a24-a157-d884de7614c8\") " pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:22:38.634956 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.634937    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcgdp\" (UniqueName: \"kubernetes.io/projected/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-kube-api-access-dcgdp\") pod \"kserve-controller-manager-7b9c77fff9-t9sh9\" (UID: \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\") " pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:22:38.750116 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.750077    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:22:38.786684 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.786652    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:22:38.893448 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.893415    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-t9sh9"]
Jul 02 14:22:38.897315 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:22:38.897286    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9bc5b8_f483_44b6_8aa9_5adfd56a0846.slice/crio-9d234c755f47345bcce5935da4bc0a07923200fa7084a5070cfb4a10cfad5f31 WatchSource:0}: Error finding container 9d234c755f47345bcce5935da4bc0a07923200fa7084a5070cfb4a10cfad5f31: Status 404 returned error can't find the container with id 9d234c755f47345bcce5935da4bc0a07923200fa7084a5070cfb4a10cfad5f31
Jul 02 14:22:38.898824 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.898803    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:22:38.925599 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:38.925572    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-dfxpj"]
Jul 02 14:22:38.928422 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:22:38.928396    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37afa26a_df0c_4a24_a157_d884de7614c8.slice/crio-52d6947f9565273c6e9653727115578da949758f625ccf3f06a6104c55db8eab WatchSource:0}: Error finding container 52d6947f9565273c6e9653727115578da949758f625ccf3f06a6104c55db8eab: Status 404 returned error can't find the container with id 52d6947f9565273c6e9653727115578da949758f625ccf3f06a6104c55db8eab
Jul 02 14:22:39.139987 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:39.139888    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9" event={"ID":"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846","Type":"ContainerStarted","Data":"9d234c755f47345bcce5935da4bc0a07923200fa7084a5070cfb4a10cfad5f31"}
Jul 02 14:22:39.140961 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:39.140942    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-dfxpj" event={"ID":"37afa26a-df0c-4a24-a157-d884de7614c8","Type":"ContainerStarted","Data":"52d6947f9565273c6e9653727115578da949758f625ccf3f06a6104c55db8eab"}
Jul 02 14:22:42.155411 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:42.155377    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9" event={"ID":"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846","Type":"ContainerStarted","Data":"a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774"}
Jul 02 14:22:42.155815 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:42.155543    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:22:42.175872 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:42.175820    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9" podStartSLOduration=1.460236744 podStartE2EDuration="4.175797471s" totalImagesPullingTime="2.715560727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:22:42.173976851 +0000 UTC m=+368.937362184" watchObservedRunningTime="2026-07-02 14:22:42.175797471 +0000 UTC m=+368.939182793"
Jul 02 14:22:43.160509 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:43.160452    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-dfxpj" event={"ID":"37afa26a-df0c-4a24-a157-d884de7614c8","Type":"ContainerStarted","Data":"bce66262f68b14917790510bad86f1198c004060d50e288ba0840e85414cd7c4"}
Jul 02 14:22:43.160916 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:43.160695    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:22:43.178381 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:43.178326    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-dfxpj" podStartSLOduration=1.388192652 podStartE2EDuration="5.178310555s" totalImagesPullingTime="3.790117903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:22:43.177625793 +0000 UTC m=+369.941011126" watchObservedRunningTime="2026-07-02 14:22:43.178310555 +0000 UTC m=+369.941695876"
Jul 02 14:22:49.165578 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:22:49.165549    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-dfxpj"
Jul 02 14:23:13.165838 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.165803    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:23:13.250031 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.250001    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-t9sh9"]
Jul 02 14:23:13.265926 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.265880    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9" podUID="4b9bc5b8-f483-44b6-8aa9-5adfd56a0846" containerName="manager" containerID="cri-o://a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774" gracePeriod=10
Jul 02 14:23:13.274223 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.274199    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-z5h9f"]
Jul 02 14:23:13.279541 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.279522    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:13.286613 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.286591    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-z5h9f"]
Jul 02 14:23:13.398168 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.398133    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0e6ea43-915a-4c51-9dcd-923b99016b90-cert\") pod \"kserve-controller-manager-7b9c77fff9-z5h9f\" (UID: \"d0e6ea43-915a-4c51-9dcd-923b99016b90\") " pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:13.398315 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.398212    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c227p\" (UniqueName: \"kubernetes.io/projected/d0e6ea43-915a-4c51-9dcd-923b99016b90-kube-api-access-c227p\") pod \"kserve-controller-manager-7b9c77fff9-z5h9f\" (UID: \"d0e6ea43-915a-4c51-9dcd-923b99016b90\") " pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:13.499118 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.499089    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0e6ea43-915a-4c51-9dcd-923b99016b90-cert\") pod \"kserve-controller-manager-7b9c77fff9-z5h9f\" (UID: \"d0e6ea43-915a-4c51-9dcd-923b99016b90\") " pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:13.499217 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.499150    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c227p\" (UniqueName: \"kubernetes.io/projected/d0e6ea43-915a-4c51-9dcd-923b99016b90-kube-api-access-c227p\") pod \"kserve-controller-manager-7b9c77fff9-z5h9f\" (UID: \"d0e6ea43-915a-4c51-9dcd-923b99016b90\") " pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:13.501528 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.501502    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0e6ea43-915a-4c51-9dcd-923b99016b90-cert\") pod \"kserve-controller-manager-7b9c77fff9-z5h9f\" (UID: \"d0e6ea43-915a-4c51-9dcd-923b99016b90\") " pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:13.504805 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.504787    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:23:13.508500 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.508466    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c227p\" (UniqueName: \"kubernetes.io/projected/d0e6ea43-915a-4c51-9dcd-923b99016b90-kube-api-access-c227p\") pod \"kserve-controller-manager-7b9c77fff9-z5h9f\" (UID: \"d0e6ea43-915a-4c51-9dcd-923b99016b90\") " pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:13.599702 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.599666    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-cert\") pod \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\" (UID: \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\") "
Jul 02 14:23:13.599858 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.599760    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcgdp\" (UniqueName: \"kubernetes.io/projected/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-kube-api-access-dcgdp\") pod \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\" (UID: \"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846\") "
Jul 02 14:23:13.601807 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.601778    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-cert" (OuterVolumeSpecName: "cert") pod "4b9bc5b8-f483-44b6-8aa9-5adfd56a0846" (UID: "4b9bc5b8-f483-44b6-8aa9-5adfd56a0846"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:23:13.601934 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.601876    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-kube-api-access-dcgdp" (OuterVolumeSpecName: "kube-api-access-dcgdp") pod "4b9bc5b8-f483-44b6-8aa9-5adfd56a0846" (UID: "4b9bc5b8-f483-44b6-8aa9-5adfd56a0846"). InnerVolumeSpecName "kube-api-access-dcgdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:23:13.636853 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.636830    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:13.700428 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.700400    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcgdp\" (UniqueName: \"kubernetes.io/projected/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-kube-api-access-dcgdp\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:23:13.700428 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.700430    2551 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846-cert\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:23:13.767125 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:13.767097    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-z5h9f"]
Jul 02 14:23:13.769927 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:23:13.769904    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e6ea43_915a_4c51_9dcd_923b99016b90.slice/crio-a0c3fcfe30a4425af3f21f5b5abf56e10a6d585de03638f438a89b88b6ba3aeb WatchSource:0}: Error finding container a0c3fcfe30a4425af3f21f5b5abf56e10a6d585de03638f438a89b88b6ba3aeb: Status 404 returned error can't find the container with id a0c3fcfe30a4425af3f21f5b5abf56e10a6d585de03638f438a89b88b6ba3aeb
Jul 02 14:23:14.269975 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.269940    2551 generic.go:356] "Generic (PLEG): container finished" podID="4b9bc5b8-f483-44b6-8aa9-5adfd56a0846" containerID="a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774" exitCode=0
Jul 02 14:23:14.270427 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.270005    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9"
Jul 02 14:23:14.270427 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.270029    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9" event={"ID":"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846","Type":"ContainerDied","Data":"a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774"}
Jul 02 14:23:14.270427 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.270100    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-t9sh9" event={"ID":"4b9bc5b8-f483-44b6-8aa9-5adfd56a0846","Type":"ContainerDied","Data":"9d234c755f47345bcce5935da4bc0a07923200fa7084a5070cfb4a10cfad5f31"}
Jul 02 14:23:14.270427 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.270126    2551 scope.go:117] "RemoveContainer" containerID="a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774"
Jul 02 14:23:14.271498 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.271449    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f" event={"ID":"d0e6ea43-915a-4c51-9dcd-923b99016b90","Type":"ContainerStarted","Data":"a0c3fcfe30a4425af3f21f5b5abf56e10a6d585de03638f438a89b88b6ba3aeb"}
Jul 02 14:23:14.284904 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.284885    2551 scope.go:117] "RemoveContainer" containerID="a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774"
Jul 02 14:23:14.285203 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:23:14.285179    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774\": container with ID starting with a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774 not found: ID does not exist" containerID="a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774"
Jul 02 14:23:14.285283 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.285214    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774"} err="failed to get container status \"a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774\": rpc error: code = NotFound desc = could not find container \"a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774\": container with ID starting with a40c438b650bd34fd35364e0d5a5f14388963a3f9e591eb33154154a5460e774 not found: ID does not exist"
Jul 02 14:23:14.293827 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.293810    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-t9sh9"]
Jul 02 14:23:14.300580 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:14.300564    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-t9sh9"]
Jul 02 14:23:15.276953 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:15.276917    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f" event={"ID":"d0e6ea43-915a-4c51-9dcd-923b99016b90","Type":"ContainerStarted","Data":"3df809c2186ccb1bbfa12b84a2cbbeaa30bc174df3b2b66ec26158d657c87406"}
Jul 02 14:23:15.277444 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:15.277067    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:15.295759 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:15.295711    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f" podStartSLOduration=1.777848047 podStartE2EDuration="2.295696619s" totalImagesPullingTime="517.848572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:23:15.294202079 +0000 UTC m=+402.057587435" watchObservedRunningTime="2026-07-02 14:23:15.295696619 +0000 UTC m=+402.059081939"
Jul 02 14:23:15.748444 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:15.748412    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b9bc5b8-f483-44b6-8aa9-5adfd56a0846" path="/var/lib/kubelet/pods/4b9bc5b8-f483-44b6-8aa9-5adfd56a0846/volumes"
Jul 02 14:23:46.285794 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:46.285753    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7b9c77fff9-z5h9f"
Jul 02 14:23:47.374192 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.374160    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-wnhvt"]
Jul 02 14:23:47.374602 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.374424    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9bc5b8-f483-44b6-8aa9-5adfd56a0846" containerName="manager"
Jul 02 14:23:47.374602 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.374435    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9bc5b8-f483-44b6-8aa9-5adfd56a0846" containerName="manager"
Jul 02 14:23:47.374602 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.374504    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9bc5b8-f483-44b6-8aa9-5adfd56a0846" containerName="manager"
Jul 02 14:23:47.377512 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.377493    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:23:47.380222 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.380189    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 02 14:23:47.380335 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.380226    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-q52n8\""
Jul 02 14:23:47.389776 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.389752    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wnhvt"]
Jul 02 14:23:47.391544 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.391523    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-bl79l"]
Jul 02 14:23:47.394596 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.394576    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:47.397381 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.397361    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 02 14:23:47.397505 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.397393    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-sks7f\""
Jul 02 14:23:47.407531 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.407509    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-bl79l"]
Jul 02 14:23:47.452714 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.452685    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5396f43d-39a7-414c-808c-52a64fc6ca1e-cert\") pod \"odh-model-controller-84b54b7f97-bl79l\" (UID: \"5396f43d-39a7-414c-808c-52a64fc6ca1e\") " pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:47.452876 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.452727    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9260c524-62d7-4478-8350-567dca7b65a8-tls-certs\") pod \"model-serving-api-fd65d7d6-wnhvt\" (UID: \"9260c524-62d7-4478-8350-567dca7b65a8\") " pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:23:47.452876 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.452750    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5cs\" (UniqueName: \"kubernetes.io/projected/5396f43d-39a7-414c-808c-52a64fc6ca1e-kube-api-access-6z5cs\") pod \"odh-model-controller-84b54b7f97-bl79l\" (UID: \"5396f43d-39a7-414c-808c-52a64fc6ca1e\") " pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:47.452876 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.452794    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fpqh\" (UniqueName: \"kubernetes.io/projected/9260c524-62d7-4478-8350-567dca7b65a8-kube-api-access-7fpqh\") pod \"model-serving-api-fd65d7d6-wnhvt\" (UID: \"9260c524-62d7-4478-8350-567dca7b65a8\") " pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:23:47.553881 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.553843    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fpqh\" (UniqueName: \"kubernetes.io/projected/9260c524-62d7-4478-8350-567dca7b65a8-kube-api-access-7fpqh\") pod \"model-serving-api-fd65d7d6-wnhvt\" (UID: \"9260c524-62d7-4478-8350-567dca7b65a8\") " pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:23:47.554074 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.553898    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5396f43d-39a7-414c-808c-52a64fc6ca1e-cert\") pod \"odh-model-controller-84b54b7f97-bl79l\" (UID: \"5396f43d-39a7-414c-808c-52a64fc6ca1e\") " pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:47.554074 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.553928    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9260c524-62d7-4478-8350-567dca7b65a8-tls-certs\") pod \"model-serving-api-fd65d7d6-wnhvt\" (UID: \"9260c524-62d7-4478-8350-567dca7b65a8\") " pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:23:47.554074 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.553945    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5cs\" (UniqueName: \"kubernetes.io/projected/5396f43d-39a7-414c-808c-52a64fc6ca1e-kube-api-access-6z5cs\") pod \"odh-model-controller-84b54b7f97-bl79l\" (UID: \"5396f43d-39a7-414c-808c-52a64fc6ca1e\") " pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:47.554074 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:23:47.554052    2551 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 02 14:23:47.554271 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:23:47.554129    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5396f43d-39a7-414c-808c-52a64fc6ca1e-cert podName:5396f43d-39a7-414c-808c-52a64fc6ca1e nodeName:}" failed. No retries permitted until 2026-07-02 14:23:48.05410828 +0000 UTC m=+434.817493596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5396f43d-39a7-414c-808c-52a64fc6ca1e-cert") pod "odh-model-controller-84b54b7f97-bl79l" (UID: "5396f43d-39a7-414c-808c-52a64fc6ca1e") : secret "odh-model-controller-webhook-cert" not found
Jul 02 14:23:47.556590 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.556559    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9260c524-62d7-4478-8350-567dca7b65a8-tls-certs\") pod \"model-serving-api-fd65d7d6-wnhvt\" (UID: \"9260c524-62d7-4478-8350-567dca7b65a8\") " pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:23:47.563557 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.563533    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fpqh\" (UniqueName: \"kubernetes.io/projected/9260c524-62d7-4478-8350-567dca7b65a8-kube-api-access-7fpqh\") pod \"model-serving-api-fd65d7d6-wnhvt\" (UID: \"9260c524-62d7-4478-8350-567dca7b65a8\") " pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:23:47.564006 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.563984    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5cs\" (UniqueName: \"kubernetes.io/projected/5396f43d-39a7-414c-808c-52a64fc6ca1e-kube-api-access-6z5cs\") pod \"odh-model-controller-84b54b7f97-bl79l\" (UID: \"5396f43d-39a7-414c-808c-52a64fc6ca1e\") " pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:47.689267 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.689179    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:23:47.819552 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:47.819526    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wnhvt"]
Jul 02 14:23:47.822019 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:23:47.821995    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9260c524_62d7_4478_8350_567dca7b65a8.slice/crio-a7de0fbf065b82c6849fb32c87df0295cfdcb6a95f1ea4df430f43220077f85c WatchSource:0}: Error finding container a7de0fbf065b82c6849fb32c87df0295cfdcb6a95f1ea4df430f43220077f85c: Status 404 returned error can't find the container with id a7de0fbf065b82c6849fb32c87df0295cfdcb6a95f1ea4df430f43220077f85c
Jul 02 14:23:48.058685 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:48.058649    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5396f43d-39a7-414c-808c-52a64fc6ca1e-cert\") pod \"odh-model-controller-84b54b7f97-bl79l\" (UID: \"5396f43d-39a7-414c-808c-52a64fc6ca1e\") " pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:48.061131 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:48.061106    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5396f43d-39a7-414c-808c-52a64fc6ca1e-cert\") pod \"odh-model-controller-84b54b7f97-bl79l\" (UID: \"5396f43d-39a7-414c-808c-52a64fc6ca1e\") " pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:48.305569 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:48.305527    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:48.407206 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:48.407020    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wnhvt" event={"ID":"9260c524-62d7-4478-8350-567dca7b65a8","Type":"ContainerStarted","Data":"a7de0fbf065b82c6849fb32c87df0295cfdcb6a95f1ea4df430f43220077f85c"}
Jul 02 14:23:48.465117 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:48.464982    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-bl79l"]
Jul 02 14:23:48.478653 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:23:48.478620    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5396f43d_39a7_414c_808c_52a64fc6ca1e.slice/crio-6255cf3aa5b465db1368e7d830c1e2c80db18577ee20a30b595847da2bde14b7 WatchSource:0}: Error finding container 6255cf3aa5b465db1368e7d830c1e2c80db18577ee20a30b595847da2bde14b7: Status 404 returned error can't find the container with id 6255cf3aa5b465db1368e7d830c1e2c80db18577ee20a30b595847da2bde14b7
Jul 02 14:23:49.413569 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:49.413497    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-bl79l" event={"ID":"5396f43d-39a7-414c-808c-52a64fc6ca1e","Type":"ContainerStarted","Data":"6255cf3aa5b465db1368e7d830c1e2c80db18577ee20a30b595847da2bde14b7"}
Jul 02 14:23:50.418895 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:50.418853    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wnhvt" event={"ID":"9260c524-62d7-4478-8350-567dca7b65a8","Type":"ContainerStarted","Data":"06f0b003b6f1aede7ac8e8dbf9f502871e154040f150cb48eb8ee4e7e6827cd3"}
Jul 02 14:23:50.419359 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:50.418972    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:23:50.439166 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:50.439106    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-wnhvt" podStartSLOduration=1.767537877 podStartE2EDuration="3.439090592s" totalImagesPullingTime="1.671552715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:23:50.437703739 +0000 UTC m=+437.201089087" watchObservedRunningTime="2026-07-02 14:23:50.439090592 +0000 UTC m=+437.202475912"
Jul 02 14:23:52.427190 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:52.427152    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-bl79l" event={"ID":"5396f43d-39a7-414c-808c-52a64fc6ca1e","Type":"ContainerStarted","Data":"859832feb835e362688ea36c1d46afd995889e3ce82c0b39112b260f3c3c8906"}
Jul 02 14:23:52.427595 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:52.427269    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:23:52.446331 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:23:52.446274    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-bl79l" podStartSLOduration=2.506236287 podStartE2EDuration="5.446258857s" totalImagesPullingTime="2.94002257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:23:52.444889431 +0000 UTC m=+439.208274753" watchObservedRunningTime="2026-07-02 14:23:52.446258857 +0000 UTC m=+439.209644225"
Jul 02 14:24:01.428829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:24:01.428754    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-wnhvt"
Jul 02 14:24:03.437125 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:24:03.437092    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-bl79l"
Jul 02 14:25:35.349432 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.349374    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"]
Jul 02 14:25:35.351941 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.351924    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.354835 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.354803    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 14:25:35.354980 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.354837    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-df2c1-predictor-serving-cert\""
Jul 02 14:25:35.354980 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.354876    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fn24b\""
Jul 02 14:25:35.356015 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.355993    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 14:25:35.356192 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.355993    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-df2c1-kube-rbac-proxy-sar-config\""
Jul 02 14:25:35.362744 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.362722    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"]
Jul 02 14:25:35.508977 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.508947    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hv5d\" (UniqueName: \"kubernetes.io/projected/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-api-access-5hv5d\") pod \"success-200-isvc-df2c1-predictor-54bbb9f954-cbnst\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") " pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.509172 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.509066    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df2c1-predictor-54bbb9f954-cbnst\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") " pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.509172 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.509105    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f35882d3-cc5b-4b91-9516-aefeb824e516-proxy-tls\") pod \"success-200-isvc-df2c1-predictor-54bbb9f954-cbnst\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") " pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.609536 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.609448    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df2c1-predictor-54bbb9f954-cbnst\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") " pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.609536 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.609507    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f35882d3-cc5b-4b91-9516-aefeb824e516-proxy-tls\") pod \"success-200-isvc-df2c1-predictor-54bbb9f954-cbnst\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") " pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.609536 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.609531    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hv5d\" (UniqueName: \"kubernetes.io/projected/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-api-access-5hv5d\") pod \"success-200-isvc-df2c1-predictor-54bbb9f954-cbnst\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") " pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.610311 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.610292    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-df2c1-predictor-54bbb9f954-cbnst\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") " pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.612245 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.612213    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f35882d3-cc5b-4b91-9516-aefeb824e516-proxy-tls\") pod \"success-200-isvc-df2c1-predictor-54bbb9f954-cbnst\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") " pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.624962 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.624935    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hv5d\" (UniqueName: \"kubernetes.io/projected/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-api-access-5hv5d\") pod \"success-200-isvc-df2c1-predictor-54bbb9f954-cbnst\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") " pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.665019 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.664978    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:35.809781 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.809750    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"]
Jul 02 14:25:35.813122 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.813103    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:35.815312 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.815287    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-df2c1-predictor-serving-cert\""
Jul 02 14:25:35.815671 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.815649    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-df2c1-kube-rbac-proxy-sar-config\""
Jul 02 14:25:35.823364 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.823337    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"]
Jul 02 14:25:35.828318 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.828251    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"]
Jul 02 14:25:35.831580 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:25:35.831550    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35882d3_cc5b_4b91_9516_aefeb824e516.slice/crio-12d3aae0057656ffb4b37d2b97f620953281093a5d721af9217267eae47bcc40 WatchSource:0}: Error finding container 12d3aae0057656ffb4b37d2b97f620953281093a5d721af9217267eae47bcc40: Status 404 returned error can't find the container with id 12d3aae0057656ffb4b37d2b97f620953281093a5d721af9217267eae47bcc40
Jul 02 14:25:35.911949 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.911782    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4z2\" (UniqueName: \"kubernetes.io/projected/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-api-access-dg4z2\") pod \"error-404-isvc-df2c1-predictor-688b4ccf47-fsdck\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") " pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:35.911949 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.911856    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-df2c1-predictor-688b4ccf47-fsdck\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") " pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:35.911949 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:35.911912    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-proxy-tls\") pod \"error-404-isvc-df2c1-predictor-688b4ccf47-fsdck\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") " pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:36.013272 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.013231    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-proxy-tls\") pod \"error-404-isvc-df2c1-predictor-688b4ccf47-fsdck\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") " pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:36.013437 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.013293    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4z2\" (UniqueName: \"kubernetes.io/projected/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-api-access-dg4z2\") pod \"error-404-isvc-df2c1-predictor-688b4ccf47-fsdck\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") " pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:36.013437 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.013346    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-df2c1-predictor-688b4ccf47-fsdck\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") " pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:36.014036 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.014013    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-df2c1-predictor-688b4ccf47-fsdck\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") " pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:36.015822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.015801    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-proxy-tls\") pod \"error-404-isvc-df2c1-predictor-688b4ccf47-fsdck\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") " pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:36.023401 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.023374    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4z2\" (UniqueName: \"kubernetes.io/projected/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-api-access-dg4z2\") pod \"error-404-isvc-df2c1-predictor-688b4ccf47-fsdck\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") " pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:36.125648 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.125598    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:36.261288 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.261255    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"]
Jul 02 14:25:36.264059 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:25:36.264031    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a1b01f_198d_4301_b0fe_388cdbbbdd46.slice/crio-6e0a5477867a728abd679ae4b5c07c7621de52ce3170f30f5277159602363153 WatchSource:0}: Error finding container 6e0a5477867a728abd679ae4b5c07c7621de52ce3170f30f5277159602363153: Status 404 returned error can't find the container with id 6e0a5477867a728abd679ae4b5c07c7621de52ce3170f30f5277159602363153
Jul 02 14:25:36.825303 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.825222    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" event={"ID":"f35882d3-cc5b-4b91-9516-aefeb824e516","Type":"ContainerStarted","Data":"12d3aae0057656ffb4b37d2b97f620953281093a5d721af9217267eae47bcc40"}
Jul 02 14:25:36.830105 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:36.830071    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" event={"ID":"e3a1b01f-198d-4301-b0fe-388cdbbbdd46","Type":"ContainerStarted","Data":"6e0a5477867a728abd679ae4b5c07c7621de52ce3170f30f5277159602363153"}
Jul 02 14:25:52.914004 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:52.913956    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" event={"ID":"e3a1b01f-198d-4301-b0fe-388cdbbbdd46","Type":"ContainerStarted","Data":"05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3"}
Jul 02 14:25:52.916483 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:52.916438    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" event={"ID":"f35882d3-cc5b-4b91-9516-aefeb824e516","Type":"ContainerStarted","Data":"702c5284611f3ff2510d2ff0cb5b7d73166c25219a0c43115636f56561411df0"}
Jul 02 14:25:54.927343 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.927306    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" event={"ID":"f35882d3-cc5b-4b91-9516-aefeb824e516","Type":"ContainerStarted","Data":"177480d5b4f865a8793ff8b226ffa5fe230df20a816f8b491b02c2f0655f753a"}
Jul 02 14:25:54.927827 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.927496    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:54.927827 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.927547    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:25:54.928937 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.928902    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 02 14:25:54.929448 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.929425    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" event={"ID":"e3a1b01f-198d-4301-b0fe-388cdbbbdd46","Type":"ContainerStarted","Data":"0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511"}
Jul 02 14:25:54.929548 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.929497    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:54.929548 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.929522    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:25:54.930546 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.930525    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 02 14:25:54.945989 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.945905    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podStartSLOduration=1.160901632 podStartE2EDuration="19.945891336s" totalImagesPullingTime="18.784989704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:25:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:25:54.94405694 +0000 UTC m=+561.707442275" watchObservedRunningTime="2026-07-02 14:25:54.945891336 +0000 UTC m=+561.709276657"
Jul 02 14:25:54.961854 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:54.961807    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podStartSLOduration=1.573921262 podStartE2EDuration="19.961779794s" totalImagesPullingTime="18.387858532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:25:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:25:54.960730975 +0000 UTC m=+561.724116296" watchObservedRunningTime="2026-07-02 14:25:54.961779794 +0000 UTC m=+561.725165115"
Jul 02 14:25:55.933056 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:55.933014    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 02 14:25:55.933426 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:25:55.933103    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 02 14:26:00.938194 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:00.938153    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:26:00.938608 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:00.938366    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:26:00.938608 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:00.938588    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 02 14:26:00.939008 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:00.938988    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 02 14:26:10.939415 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:10.939374    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 02 14:26:10.939415 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:10.939374    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 02 14:26:20.938599 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:20.938559    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 02 14:26:20.938992 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:20.938972    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 02 14:26:30.939368 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:30.939329    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 02 14:26:30.939761 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:30.939330    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 02 14:26:33.722732 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:33.722707    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:26:33.724386 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:33.724361    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:26:40.939208 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:40.939175    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:26:40.940460 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:26:40.939595    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:27:09.470321 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.470241    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"]
Jul 02 14:27:09.474916 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.474896    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.477564 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.477542    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1c2b7-kube-rbac-proxy-sar-config\""
Jul 02 14:27:09.477678 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.477572    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1c2b7-predictor-serving-cert\""
Jul 02 14:27:09.484953 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.484931    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"]
Jul 02 14:27:09.503057 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.503034    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"]
Jul 02 14:27:09.503325 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.503296    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container" containerID="cri-o://702c5284611f3ff2510d2ff0cb5b7d73166c25219a0c43115636f56561411df0" gracePeriod=30
Jul 02 14:27:09.503617 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.503354    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kube-rbac-proxy" containerID="cri-o://177480d5b4f865a8793ff8b226ffa5fe230df20a816f8b491b02c2f0655f753a" gracePeriod=30
Jul 02 14:27:09.556811 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.556779    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"]
Jul 02 14:27:09.558666 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.558639    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqfrk\" (UniqueName: \"kubernetes.io/projected/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-api-access-qqfrk\") pod \"success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") " pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.558813 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.558713    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-proxy-tls\") pod \"success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") " pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.558813 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.558750    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") " pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.560187 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.560166    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.563180 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.563163    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1c2b7-kube-rbac-proxy-sar-config\""
Jul 02 14:27:09.563304 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.563223    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1c2b7-predictor-serving-cert\""
Jul 02 14:27:09.569764 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.569741    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"]
Jul 02 14:27:09.597234 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.597210    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"]
Jul 02 14:27:09.597519 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.597490    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container" containerID="cri-o://05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3" gracePeriod=30
Jul 02 14:27:09.597637 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.597528    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kube-rbac-proxy" containerID="cri-o://0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511" gracePeriod=30
Jul 02 14:27:09.659030 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.658998    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") " pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.659194 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.659039    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55158916-8aa3-4215-9c83-02b1224d1a62-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") " pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.659194 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.659058    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55158916-8aa3-4215-9c83-02b1224d1a62-proxy-tls\") pod \"error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") " pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.659194 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.659081    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqfrk\" (UniqueName: \"kubernetes.io/projected/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-api-access-qqfrk\") pod \"success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") " pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.659194 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.659098    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvc9q\" (UniqueName: \"kubernetes.io/projected/55158916-8aa3-4215-9c83-02b1224d1a62-kube-api-access-tvc9q\") pod \"error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") " pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.659348 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.659227    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-proxy-tls\") pod \"success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") " pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.659643 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.659623    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") " pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.661503 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.661456    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-proxy-tls\") pod \"success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") " pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.669278 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.669250    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqfrk\" (UniqueName: \"kubernetes.io/projected/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-api-access-qqfrk\") pod \"success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") " pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.759755 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.759724    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55158916-8aa3-4215-9c83-02b1224d1a62-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") " pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.759908 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.759762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55158916-8aa3-4215-9c83-02b1224d1a62-proxy-tls\") pod \"error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") " pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.759908 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.759797    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvc9q\" (UniqueName: \"kubernetes.io/projected/55158916-8aa3-4215-9c83-02b1224d1a62-kube-api-access-tvc9q\") pod \"error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") " pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.760335 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.760304    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55158916-8aa3-4215-9c83-02b1224d1a62-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") " pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.762127 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.762097    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55158916-8aa3-4215-9c83-02b1224d1a62-proxy-tls\") pod \"error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") " pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.768906 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.768885    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvc9q\" (UniqueName: \"kubernetes.io/projected/55158916-8aa3-4215-9c83-02b1224d1a62-kube-api-access-tvc9q\") pod \"error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") " pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.788258 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.788230    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:09.872805 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.872777    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:09.919212 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:09.918305    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"]
Jul 02 14:27:09.920922 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:27:09.920887    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942cb8ee_ae7e_412b_a8ed_c2a5b3628019.slice/crio-e7677342cbe05b26632973caa93c113731e9a305856aaadcb2cba6f0e592c45c WatchSource:0}: Error finding container e7677342cbe05b26632973caa93c113731e9a305856aaadcb2cba6f0e592c45c: Status 404 returned error can't find the container with id e7677342cbe05b26632973caa93c113731e9a305856aaadcb2cba6f0e592c45c
Jul 02 14:27:10.016512 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:27:10.013729    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55158916_8aa3_4215_9c83_02b1224d1a62.slice/crio-a24c9ddf32e0eee07b02ade0ee538572cc71260169f4cb036fcadd056c441a09 WatchSource:0}: Error finding container a24c9ddf32e0eee07b02ade0ee538572cc71260169f4cb036fcadd056c441a09: Status 404 returned error can't find the container with id a24c9ddf32e0eee07b02ade0ee538572cc71260169f4cb036fcadd056c441a09
Jul 02 14:27:10.016512 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.013877    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"]
Jul 02 14:27:10.219143 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.219105    2551 generic.go:356] "Generic (PLEG): container finished" podID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerID="0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511" exitCode=2
Jul 02 14:27:10.219318 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.219174    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" event={"ID":"e3a1b01f-198d-4301-b0fe-388cdbbbdd46","Type":"ContainerDied","Data":"0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511"}
Jul 02 14:27:10.220907 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.220880    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" event={"ID":"55158916-8aa3-4215-9c83-02b1224d1a62","Type":"ContainerStarted","Data":"8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3"}
Jul 02 14:27:10.221027 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.220936    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" event={"ID":"55158916-8aa3-4215-9c83-02b1224d1a62","Type":"ContainerStarted","Data":"79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31"}
Jul 02 14:27:10.221027 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.220974    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:10.221027 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.221013    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" event={"ID":"55158916-8aa3-4215-9c83-02b1224d1a62","Type":"ContainerStarted","Data":"a24c9ddf32e0eee07b02ade0ee538572cc71260169f4cb036fcadd056c441a09"}
Jul 02 14:27:10.221202 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.221063    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:10.222781 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.222757    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 14:27:10.223044 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.223024    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" event={"ID":"942cb8ee-ae7e-412b-a8ed-c2a5b3628019","Type":"ContainerStarted","Data":"d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e"}
Jul 02 14:27:10.223143 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.223073    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:10.223143 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.223107    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:10.223143 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.223130    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" event={"ID":"942cb8ee-ae7e-412b-a8ed-c2a5b3628019","Type":"ContainerStarted","Data":"2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9"}
Jul 02 14:27:10.223286 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.223151    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" event={"ID":"942cb8ee-ae7e-412b-a8ed-c2a5b3628019","Type":"ContainerStarted","Data":"e7677342cbe05b26632973caa93c113731e9a305856aaadcb2cba6f0e592c45c"}
Jul 02 14:27:10.224152 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.224125    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 02 14:27:10.224744 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.224725    2551 generic.go:356] "Generic (PLEG): container finished" podID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerID="177480d5b4f865a8793ff8b226ffa5fe230df20a816f8b491b02c2f0655f753a" exitCode=2
Jul 02 14:27:10.224803 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.224761    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" event={"ID":"f35882d3-cc5b-4b91-9516-aefeb824e516","Type":"ContainerDied","Data":"177480d5b4f865a8793ff8b226ffa5fe230df20a816f8b491b02c2f0655f753a"}
Jul 02 14:27:10.242853 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.242807    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podStartSLOduration=1.242790281 podStartE2EDuration="1.242790281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:27:10.239761237 +0000 UTC m=+637.003146556" watchObservedRunningTime="2026-07-02 14:27:10.242790281 +0000 UTC m=+637.006175603"
Jul 02 14:27:10.259773 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.259733    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podStartSLOduration=1.259723938 podStartE2EDuration="1.259723938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:27:10.257659067 +0000 UTC m=+637.021044389" watchObservedRunningTime="2026-07-02 14:27:10.259723938 +0000 UTC m=+637.023109257"
Jul 02 14:27:10.933455 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.933409    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 02 14:27:10.933820 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.933410    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 02 14:27:10.939390 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.939360    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 02 14:27:10.939539 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:10.939360    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 02 14:27:11.229706 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:11.229610    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 02 14:27:11.229896 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:11.229749    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 14:27:12.943525 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:12.943501    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:27:12.978988 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:12.978955    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-proxy-tls\") pod \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") "
Jul 02 14:27:12.979147 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:12.979013    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg4z2\" (UniqueName: \"kubernetes.io/projected/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-api-access-dg4z2\") pod \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") "
Jul 02 14:27:12.979147 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:12.979057    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-rbac-proxy-sar-config\") pod \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\" (UID: \"e3a1b01f-198d-4301-b0fe-388cdbbbdd46\") "
Jul 02 14:27:12.979464 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:12.979438    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3a1b01f-198d-4301-b0fe-388cdbbbdd46" (UID: "e3a1b01f-198d-4301-b0fe-388cdbbbdd46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:27:12.981083 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:12.981034    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3a1b01f-198d-4301-b0fe-388cdbbbdd46" (UID: "e3a1b01f-198d-4301-b0fe-388cdbbbdd46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:27:12.981083 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:12.981076    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-api-access-dg4z2" (OuterVolumeSpecName: "kube-api-access-dg4z2") pod "e3a1b01f-198d-4301-b0fe-388cdbbbdd46" (UID: "e3a1b01f-198d-4301-b0fe-388cdbbbdd46"). InnerVolumeSpecName "kube-api-access-dg4z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:27:13.080189 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.080112    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:27:13.080189 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.080138    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg4z2\" (UniqueName: \"kubernetes.io/projected/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-api-access-dg4z2\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:27:13.080189 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.080146    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a1b01f-198d-4301-b0fe-388cdbbbdd46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:27:13.239300 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.239270    2551 generic.go:356] "Generic (PLEG): container finished" podID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerID="702c5284611f3ff2510d2ff0cb5b7d73166c25219a0c43115636f56561411df0" exitCode=0
Jul 02 14:27:13.239444 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.239332    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" event={"ID":"f35882d3-cc5b-4b91-9516-aefeb824e516","Type":"ContainerDied","Data":"702c5284611f3ff2510d2ff0cb5b7d73166c25219a0c43115636f56561411df0"}
Jul 02 14:27:13.240868 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.240839    2551 generic.go:356] "Generic (PLEG): container finished" podID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerID="05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3" exitCode=0
Jul 02 14:27:13.240959 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.240871    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" event={"ID":"e3a1b01f-198d-4301-b0fe-388cdbbbdd46","Type":"ContainerDied","Data":"05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3"}
Jul 02 14:27:13.240959 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.240885    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"
Jul 02 14:27:13.240959 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.240920    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck" event={"ID":"e3a1b01f-198d-4301-b0fe-388cdbbbdd46","Type":"ContainerDied","Data":"6e0a5477867a728abd679ae4b5c07c7621de52ce3170f30f5277159602363153"}
Jul 02 14:27:13.240959 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.240949    2551 scope.go:117] "RemoveContainer" containerID="0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511"
Jul 02 14:27:13.242306 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.242288    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:27:13.249161 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.249145    2551 scope.go:117] "RemoveContainer" containerID="05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3"
Jul 02 14:27:13.256243 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.256226    2551 scope.go:117] "RemoveContainer" containerID="0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511"
Jul 02 14:27:13.256494 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:27:13.256457    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511\": container with ID starting with 0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511 not found: ID does not exist" containerID="0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511"
Jul 02 14:27:13.256580 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.256499    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511"} err="failed to get container status \"0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511\": rpc error: code = NotFound desc = could not find container \"0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511\": container with ID starting with 0851c6fdfafe68c4e4712b89d8883b7439c12b705165554c49ea8fad02f3e511 not found: ID does not exist"
Jul 02 14:27:13.256580 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.256517    2551 scope.go:117] "RemoveContainer" containerID="05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3"
Jul 02 14:27:13.256747 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:27:13.256733    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3\": container with ID starting with 05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3 not found: ID does not exist" containerID="05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3"
Jul 02 14:27:13.256779 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.256751    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3"} err="failed to get container status \"05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3\": rpc error: code = NotFound desc = could not find container \"05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3\": container with ID starting with 05cc9ea03ee85bde1a50917bc95142b4de14ccdb3b4e53ebb19471cd8ac57fd3 not found: ID does not exist"
Jul 02 14:27:13.280526 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.280503    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"]
Jul 02 14:27:13.282065 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.282050    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hv5d\" (UniqueName: \"kubernetes.io/projected/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-api-access-5hv5d\") pod \"f35882d3-cc5b-4b91-9516-aefeb824e516\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") "
Jul 02 14:27:13.282123 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.282078    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-rbac-proxy-sar-config\") pod \"f35882d3-cc5b-4b91-9516-aefeb824e516\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") "
Jul 02 14:27:13.282158 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.282138    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f35882d3-cc5b-4b91-9516-aefeb824e516-proxy-tls\") pod \"f35882d3-cc5b-4b91-9516-aefeb824e516\" (UID: \"f35882d3-cc5b-4b91-9516-aefeb824e516\") "
Jul 02 14:27:13.282493 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.282454    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f35882d3-cc5b-4b91-9516-aefeb824e516" (UID: "f35882d3-cc5b-4b91-9516-aefeb824e516"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:27:13.284513 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.284486    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-api-access-5hv5d" (OuterVolumeSpecName: "kube-api-access-5hv5d") pod "f35882d3-cc5b-4b91-9516-aefeb824e516" (UID: "f35882d3-cc5b-4b91-9516-aefeb824e516"). InnerVolumeSpecName "kube-api-access-5hv5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:27:13.284513 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.284499    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f35882d3-cc5b-4b91-9516-aefeb824e516-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f35882d3-cc5b-4b91-9516-aefeb824e516" (UID: "f35882d3-cc5b-4b91-9516-aefeb824e516"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:27:13.286703 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.286678    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-df2c1-predictor-688b4ccf47-fsdck"]
Jul 02 14:27:13.383019 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.382931    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f35882d3-cc5b-4b91-9516-aefeb824e516-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:27:13.383019 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.382968    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hv5d\" (UniqueName: \"kubernetes.io/projected/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-api-access-5hv5d\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:27:13.383019 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.382977    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f35882d3-cc5b-4b91-9516-aefeb824e516-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:27:13.748977 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:13.748946    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" path="/var/lib/kubelet/pods/e3a1b01f-198d-4301-b0fe-388cdbbbdd46/volumes"
Jul 02 14:27:14.246389 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:14.246368    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"
Jul 02 14:27:14.246776 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:14.246366    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst" event={"ID":"f35882d3-cc5b-4b91-9516-aefeb824e516","Type":"ContainerDied","Data":"12d3aae0057656ffb4b37d2b97f620953281093a5d721af9217267eae47bcc40"}
Jul 02 14:27:14.246776 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:14.246529    2551 scope.go:117] "RemoveContainer" containerID="177480d5b4f865a8793ff8b226ffa5fe230df20a816f8b491b02c2f0655f753a"
Jul 02 14:27:14.254435 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:14.254417    2551 scope.go:117] "RemoveContainer" containerID="702c5284611f3ff2510d2ff0cb5b7d73166c25219a0c43115636f56561411df0"
Jul 02 14:27:14.269809 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:14.269786    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"]
Jul 02 14:27:14.274203 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:14.274185    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-df2c1-predictor-54bbb9f954-cbnst"]
Jul 02 14:27:15.748153 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:15.748115    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" path="/var/lib/kubelet/pods/f35882d3-cc5b-4b91-9516-aefeb824e516/volumes"
Jul 02 14:27:16.233728 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:16.233605    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:16.234558 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:16.234513    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 14:27:16.235402 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:16.235382    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:27:16.235782 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:16.235765    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 02 14:27:26.234540 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:26.234502    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 14:27:26.236714 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:26.236689    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 02 14:27:36.235108 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:36.235068    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 14:27:36.236290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:36.236266    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 02 14:27:46.234812 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:46.234773    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 14:27:46.235994 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:46.235972    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 02 14:27:56.235911 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:56.235882    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:27:56.236630 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:27:56.236609    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:28:05.427599 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.427564    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"]
Jul 02 14:28:05.428104 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428050    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kube-rbac-proxy"
Jul 02 14:28:05.428104 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428064    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kube-rbac-proxy"
Jul 02 14:28:05.428104 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428077    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container"
Jul 02 14:28:05.428104 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428085    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container"
Jul 02 14:28:05.428104 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428097    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kube-rbac-proxy"
Jul 02 14:28:05.428104 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428106    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kube-rbac-proxy"
Jul 02 14:28:05.428283 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428120    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container"
Jul 02 14:28:05.428283 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428128    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container"
Jul 02 14:28:05.428283 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428195    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kube-rbac-proxy"
Jul 02 14:28:05.428283 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428206    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kube-rbac-proxy"
Jul 02 14:28:05.428283 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428216    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f35882d3-cc5b-4b91-9516-aefeb824e516" containerName="kserve-container"
Jul 02 14:28:05.428283 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.428224    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a1b01f-198d-4301-b0fe-388cdbbbdd46" containerName="kserve-container"
Jul 02 14:28:05.431974 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.431952    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.434901 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.434884    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9a7ca-kube-rbac-proxy-sar-config\""
Jul 02 14:28:05.435154 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.435138    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9a7ca-predictor-serving-cert\""
Jul 02 14:28:05.445295 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.445274    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"]
Jul 02 14:28:05.566250 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.566216    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"]
Jul 02 14:28:05.569533 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.569517    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.575548 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.575528    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9a7ca-predictor-serving-cert\""
Jul 02 14:28:05.576300 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.576273    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9a7ca-kube-rbac-proxy-sar-config\""
Jul 02 14:28:05.582267 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.582239    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"]
Jul 02 14:28:05.601516 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.601491    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-proxy-tls\") pod \"success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") " pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.601634 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.601546    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwpt\" (UniqueName: \"kubernetes.io/projected/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-api-access-2xwpt\") pod \"success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") " pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.601634 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.601566    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") " pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.702140 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.702058    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwpt\" (UniqueName: \"kubernetes.io/projected/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-api-access-2xwpt\") pod \"success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") " pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.702140 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.702098    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") " pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.702140 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.702120    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") " pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.702397 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.702157    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-proxy-tls\") pod \"error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") " pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.702397 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.702179    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmtsv\" (UniqueName: \"kubernetes.io/projected/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-api-access-qmtsv\") pod \"error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") " pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.702397 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.702206    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-proxy-tls\") pod \"success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") " pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.702825 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.702800    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") " pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.704690 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.704660    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-proxy-tls\") pod \"success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") " pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.713541 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.713322    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwpt\" (UniqueName: \"kubernetes.io/projected/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-api-access-2xwpt\") pod \"success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") " pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.743182 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.743146    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:05.802868 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.802684    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") " pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.803007 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.802928    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-proxy-tls\") pod \"error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") " pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.803007 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.802963    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmtsv\" (UniqueName: \"kubernetes.io/projected/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-api-access-qmtsv\") pod \"error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") " pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.803464 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.803409    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") " pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.805233 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.805210    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-proxy-tls\") pod \"error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") " pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.816138 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.816108    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmtsv\" (UniqueName: \"kubernetes.io/projected/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-api-access-qmtsv\") pod \"error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") " pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.879802 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.879768    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:05.884360 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:05.884233    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"]
Jul 02 14:28:05.886675 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:28:05.886649    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7a0abb_8b1f_4132_b57f_ba9437271dbd.slice/crio-e20ccc4d1d593bbd34b80e9d096a9fd263d8795f5a759d158796c48a2d86111f WatchSource:0}: Error finding container e20ccc4d1d593bbd34b80e9d096a9fd263d8795f5a759d158796c48a2d86111f: Status 404 returned error can't find the container with id e20ccc4d1d593bbd34b80e9d096a9fd263d8795f5a759d158796c48a2d86111f
Jul 02 14:28:06.023039 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.023016    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"]
Jul 02 14:28:06.024752 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:28:06.024726    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d71ebfa_3c34_40a6_9373_36a7a7a9a19c.slice/crio-0e3b8ebed6e664c4bea1de8c1614c5fadfc461cb67e765e7c32a3bcbf5bce211 WatchSource:0}: Error finding container 0e3b8ebed6e664c4bea1de8c1614c5fadfc461cb67e765e7c32a3bcbf5bce211: Status 404 returned error can't find the container with id 0e3b8ebed6e664c4bea1de8c1614c5fadfc461cb67e765e7c32a3bcbf5bce211
Jul 02 14:28:06.451834 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.451794    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" event={"ID":"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c","Type":"ContainerStarted","Data":"ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8"}
Jul 02 14:28:06.452255 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.451902    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:06.452255 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.451934    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:06.452255 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.451961    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" event={"ID":"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c","Type":"ContainerStarted","Data":"7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e"}
Jul 02 14:28:06.452255 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.451982    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" event={"ID":"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c","Type":"ContainerStarted","Data":"0e3b8ebed6e664c4bea1de8c1614c5fadfc461cb67e765e7c32a3bcbf5bce211"}
Jul 02 14:28:06.453575 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.453544    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" event={"ID":"9f7a0abb-8b1f-4132-b57f-ba9437271dbd","Type":"ContainerStarted","Data":"776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca"}
Jul 02 14:28:06.453673 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.453603    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" event={"ID":"9f7a0abb-8b1f-4132-b57f-ba9437271dbd","Type":"ContainerStarted","Data":"a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5"}
Jul 02 14:28:06.453673 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.453634    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" event={"ID":"9f7a0abb-8b1f-4132-b57f-ba9437271dbd","Type":"ContainerStarted","Data":"e20ccc4d1d593bbd34b80e9d096a9fd263d8795f5a759d158796c48a2d86111f"}
Jul 02 14:28:06.453673 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.453632    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 14:28:06.453673 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.453669    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:06.476851 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.476760    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podStartSLOduration=1.476746694 podStartE2EDuration="1.476746694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:28:06.476682219 +0000 UTC m=+693.240067543" watchObservedRunningTime="2026-07-02 14:28:06.476746694 +0000 UTC m=+693.240132012"
Jul 02 14:28:06.501763 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:06.501716    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podStartSLOduration=1.5017020410000002 podStartE2EDuration="1.501702041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:28:06.50143516 +0000 UTC m=+693.264820481" watchObservedRunningTime="2026-07-02 14:28:06.501702041 +0000 UTC m=+693.265087362"
Jul 02 14:28:07.458203 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:07.458172    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:07.458694 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:07.458239    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 14:28:07.459495 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:07.459448    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 14:28:08.461976 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:08.461938    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 14:28:12.463016 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:12.462990    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:12.463459 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:12.463430    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 14:28:13.466875 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:13.466848    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:28:13.467352 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:13.467324    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 14:28:22.464116 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:22.464074    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 14:28:23.468123 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:23.468086    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 14:28:32.463732 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:32.463690    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 14:28:33.468118 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:33.468076    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 14:28:42.463512 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:42.463452    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 14:28:43.468309 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:43.468271    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 14:28:52.464093 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:52.464063    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:28:53.468312 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:28:53.468284    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:31:33.753924 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:31:33.753896    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:31:33.754772 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:31:33.754752    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:36:24.421737 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.421650    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"]
Jul 02 14:36:24.422258 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.421942    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container" containerID="cri-o://2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9" gracePeriod=30
Jul 02 14:36:24.422258 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.421988    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kube-rbac-proxy" containerID="cri-o://d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e" gracePeriod=30
Jul 02 14:36:24.471107 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.471074    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"]
Jul 02 14:36:24.474815 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.474794    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.477159 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.477136    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fbc8b-predictor-serving-cert\""
Jul 02 14:36:24.477284 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.477172    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fbc8b-kube-rbac-proxy-sar-config\""
Jul 02 14:36:24.484609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.484586    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"]
Jul 02 14:36:24.566013 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.565967    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"]
Jul 02 14:36:24.569406 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.569386    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:24.571837 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.571816    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fbc8b-predictor-serving-cert\""
Jul 02 14:36:24.571966 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.571887    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fbc8b-kube-rbac-proxy-sar-config\""
Jul 02 14:36:24.578364 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.578343    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"]
Jul 02 14:36:24.582352 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.582176    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hvn\" (UniqueName: \"kubernetes.io/projected/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-api-access-82hvn\") pod \"success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.582352 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.582228    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0e9c74-f5b3-407c-a5d5-15377a551448-proxy-tls\") pod \"success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.582352 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.582291    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.618205 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.618171    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"]
Jul 02 14:36:24.618526 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.618449    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container" containerID="cri-o://79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31" gracePeriod=30
Jul 02 14:36:24.618526 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.618484    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kube-rbac-proxy" containerID="cri-o://8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3" gracePeriod=30
Jul 02 14:36:24.683604 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.683509    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.683604 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.683551    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae07db8f-0d68-470b-afaa-b5cee9315f7e-proxy-tls\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:24.683604 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.683592    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hvn\" (UniqueName: \"kubernetes.io/projected/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-api-access-82hvn\") pod \"success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.683877 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.683613    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prx2s\" (UniqueName: \"kubernetes.io/projected/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-api-access-prx2s\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:24.683877 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.683634    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0e9c74-f5b3-407c-a5d5-15377a551448-proxy-tls\") pod \"success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.683877 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.683670    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:24.684318 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.684291    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.686402 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.686374    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0e9c74-f5b3-407c-a5d5-15377a551448-proxy-tls\") pod \"success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.693200 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.693149    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hvn\" (UniqueName: \"kubernetes.io/projected/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-api-access-82hvn\") pod \"success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") " pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.785125 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.785090    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:24.785290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.785140    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae07db8f-0d68-470b-afaa-b5cee9315f7e-proxy-tls\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:24.785290 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:36:24.785244    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-serving-cert: secret "error-404-isvc-fbc8b-predictor-serving-cert" not found
Jul 02 14:36:24.785290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.785254    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prx2s\" (UniqueName: \"kubernetes.io/projected/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-api-access-prx2s\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:24.785411 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:36:24.785301    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae07db8f-0d68-470b-afaa-b5cee9315f7e-proxy-tls podName:ae07db8f-0d68-470b-afaa-b5cee9315f7e nodeName:}" failed. No retries permitted until 2026-07-02 14:36:25.285283742 +0000 UTC m=+1192.048669055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae07db8f-0d68-470b-afaa-b5cee9315f7e-proxy-tls") pod "error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" (UID: "ae07db8f-0d68-470b-afaa-b5cee9315f7e") : secret "error-404-isvc-fbc8b-predictor-serving-cert" not found
Jul 02 14:36:24.785929 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.785907    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:24.787668 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.787638    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:24.795228 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.795199    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prx2s\" (UniqueName: \"kubernetes.io/projected/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-api-access-prx2s\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:24.927213 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:24.927184    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"]
Jul 02 14:36:24.929314 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:36:24.929288    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0e9c74_f5b3_407c_a5d5_15377a551448.slice/crio-5d8f5c47f308dd9f8458567d531769fbea30535a8a31197d63a16b09aabc9e8d WatchSource:0}: Error finding container 5d8f5c47f308dd9f8458567d531769fbea30535a8a31197d63a16b09aabc9e8d: Status 404 returned error can't find the container with id 5d8f5c47f308dd9f8458567d531769fbea30535a8a31197d63a16b09aabc9e8d
Jul 02 14:36:25.291486 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.291418    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae07db8f-0d68-470b-afaa-b5cee9315f7e-proxy-tls\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:25.294109 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.294075    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae07db8f-0d68-470b-afaa-b5cee9315f7e-proxy-tls\") pod \"error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") " pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:25.394211 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.394072    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" event={"ID":"3a0e9c74-f5b3-407c-a5d5-15377a551448","Type":"ContainerStarted","Data":"d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855"}
Jul 02 14:36:25.394211 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.394155    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" event={"ID":"3a0e9c74-f5b3-407c-a5d5-15377a551448","Type":"ContainerStarted","Data":"6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a"}
Jul 02 14:36:25.394211 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.394190    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" event={"ID":"3a0e9c74-f5b3-407c-a5d5-15377a551448","Type":"ContainerStarted","Data":"5d8f5c47f308dd9f8458567d531769fbea30535a8a31197d63a16b09aabc9e8d"}
Jul 02 14:36:25.394508 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.394370    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:25.394508 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.394432    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:25.395782 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.395753    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 14:36:25.396332 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.396310    2551 generic.go:356] "Generic (PLEG): container finished" podID="55158916-8aa3-4215-9c83-02b1224d1a62" containerID="8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3" exitCode=2
Jul 02 14:36:25.396410 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.396377    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" event={"ID":"55158916-8aa3-4215-9c83-02b1224d1a62","Type":"ContainerDied","Data":"8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3"}
Jul 02 14:36:25.398058 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.398035    2551 generic.go:356] "Generic (PLEG): container finished" podID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerID="d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e" exitCode=2
Jul 02 14:36:25.398153 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.398096    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" event={"ID":"942cb8ee-ae7e-412b-a8ed-c2a5b3628019","Type":"ContainerDied","Data":"d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e"}
Jul 02 14:36:25.419500 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.419423    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podStartSLOduration=1.41940595 podStartE2EDuration="1.41940595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:36:25.417190851 +0000 UTC m=+1192.180576173" watchObservedRunningTime="2026-07-02 14:36:25.41940595 +0000 UTC m=+1192.182791270"
Jul 02 14:36:25.482923 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.482890    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:25.617345 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:25.617315    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"]
Jul 02 14:36:25.620597 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:36:25.620565    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae07db8f_0d68_470b_afaa_b5cee9315f7e.slice/crio-387865ba6e7959fe9c81409440266379f3c42156fdfda1c41c37f7a399f1ba10 WatchSource:0}: Error finding container 387865ba6e7959fe9c81409440266379f3c42156fdfda1c41c37f7a399f1ba10: Status 404 returned error can't find the container with id 387865ba6e7959fe9c81409440266379f3c42156fdfda1c41c37f7a399f1ba10
Jul 02 14:36:26.230005 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.229965    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 02 14:36:26.230174 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.229972    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 02 14:36:26.235390 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.235366    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 02 14:36:26.235978 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.235958    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 02 14:36:26.404659 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.404624    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" event={"ID":"ae07db8f-0d68-470b-afaa-b5cee9315f7e","Type":"ContainerStarted","Data":"3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2"}
Jul 02 14:36:26.404848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.404702    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:26.404848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.404731    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:26.404848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.404756    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" event={"ID":"ae07db8f-0d68-470b-afaa-b5cee9315f7e","Type":"ContainerStarted","Data":"a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006"}
Jul 02 14:36:26.404848 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.404783    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" event={"ID":"ae07db8f-0d68-470b-afaa-b5cee9315f7e","Type":"ContainerStarted","Data":"387865ba6e7959fe9c81409440266379f3c42156fdfda1c41c37f7a399f1ba10"}
Jul 02 14:36:26.405049 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.405021    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 14:36:26.406187 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.406161    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 14:36:26.425267 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:26.425226    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podStartSLOduration=2.425213467 podStartE2EDuration="2.425213467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:36:26.424068601 +0000 UTC m=+1193.187453922" watchObservedRunningTime="2026-07-02 14:36:26.425213467 +0000 UTC m=+1193.188598787"
Jul 02 14:36:27.409947 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:27.409901    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 14:36:27.875155 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:27.875127    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:36:27.915102 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:27.915077    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-proxy-tls\") pod \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") "
Jul 02 14:36:27.915205 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:27.915113    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-rbac-proxy-sar-config\") pod \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") "
Jul 02 14:36:27.915205 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:27.915161    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqfrk\" (UniqueName: \"kubernetes.io/projected/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-api-access-qqfrk\") pod \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\" (UID: \"942cb8ee-ae7e-412b-a8ed-c2a5b3628019\") "
Jul 02 14:36:27.915482 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:27.915447    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "942cb8ee-ae7e-412b-a8ed-c2a5b3628019" (UID: "942cb8ee-ae7e-412b-a8ed-c2a5b3628019"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:36:27.917171 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:27.917143    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "942cb8ee-ae7e-412b-a8ed-c2a5b3628019" (UID: "942cb8ee-ae7e-412b-a8ed-c2a5b3628019"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:36:27.917171 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:27.917142    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-api-access-qqfrk" (OuterVolumeSpecName: "kube-api-access-qqfrk") pod "942cb8ee-ae7e-412b-a8ed-c2a5b3628019" (UID: "942cb8ee-ae7e-412b-a8ed-c2a5b3628019"). InnerVolumeSpecName "kube-api-access-qqfrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:36:28.016708 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.016676    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:36:28.016708 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.016708    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:36:28.016873 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.016720    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqfrk\" (UniqueName: \"kubernetes.io/projected/942cb8ee-ae7e-412b-a8ed-c2a5b3628019-kube-api-access-qqfrk\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:36:28.054187 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.054163    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:36:28.117249 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.117214    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55158916-8aa3-4215-9c83-02b1224d1a62-proxy-tls\") pod \"55158916-8aa3-4215-9c83-02b1224d1a62\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") "
Jul 02 14:36:28.117398 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.117256    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55158916-8aa3-4215-9c83-02b1224d1a62-kube-rbac-proxy-sar-config\") pod \"55158916-8aa3-4215-9c83-02b1224d1a62\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") "
Jul 02 14:36:28.117398 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.117295    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvc9q\" (UniqueName: \"kubernetes.io/projected/55158916-8aa3-4215-9c83-02b1224d1a62-kube-api-access-tvc9q\") pod \"55158916-8aa3-4215-9c83-02b1224d1a62\" (UID: \"55158916-8aa3-4215-9c83-02b1224d1a62\") "
Jul 02 14:36:28.117677 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.117655    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55158916-8aa3-4215-9c83-02b1224d1a62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55158916-8aa3-4215-9c83-02b1224d1a62" (UID: "55158916-8aa3-4215-9c83-02b1224d1a62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:36:28.119385 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.119349    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55158916-8aa3-4215-9c83-02b1224d1a62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55158916-8aa3-4215-9c83-02b1224d1a62" (UID: "55158916-8aa3-4215-9c83-02b1224d1a62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:36:28.119385 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.119374    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55158916-8aa3-4215-9c83-02b1224d1a62-kube-api-access-tvc9q" (OuterVolumeSpecName: "kube-api-access-tvc9q") pod "55158916-8aa3-4215-9c83-02b1224d1a62" (UID: "55158916-8aa3-4215-9c83-02b1224d1a62"). InnerVolumeSpecName "kube-api-access-tvc9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:36:28.218371 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.218273    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55158916-8aa3-4215-9c83-02b1224d1a62-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:36:28.218371 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.218315    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55158916-8aa3-4215-9c83-02b1224d1a62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:36:28.218371 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.218327    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvc9q\" (UniqueName: \"kubernetes.io/projected/55158916-8aa3-4215-9c83-02b1224d1a62-kube-api-access-tvc9q\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:36:28.415457 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.415423    2551 generic.go:356] "Generic (PLEG): container finished" podID="55158916-8aa3-4215-9c83-02b1224d1a62" containerID="79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31" exitCode=0
Jul 02 14:36:28.415893 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.415487    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"
Jul 02 14:36:28.415893 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.415512    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" event={"ID":"55158916-8aa3-4215-9c83-02b1224d1a62","Type":"ContainerDied","Data":"79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31"}
Jul 02 14:36:28.415893 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.415572    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc" event={"ID":"55158916-8aa3-4215-9c83-02b1224d1a62","Type":"ContainerDied","Data":"a24c9ddf32e0eee07b02ade0ee538572cc71260169f4cb036fcadd056c441a09"}
Jul 02 14:36:28.415893 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.415602    2551 scope.go:117] "RemoveContainer" containerID="8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3"
Jul 02 14:36:28.417285 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.417259    2551 generic.go:356] "Generic (PLEG): container finished" podID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerID="2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9" exitCode=0
Jul 02 14:36:28.417399 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.417315    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"
Jul 02 14:36:28.417399 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.417325    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" event={"ID":"942cb8ee-ae7e-412b-a8ed-c2a5b3628019","Type":"ContainerDied","Data":"2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9"}
Jul 02 14:36:28.417399 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.417375    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4" event={"ID":"942cb8ee-ae7e-412b-a8ed-c2a5b3628019","Type":"ContainerDied","Data":"e7677342cbe05b26632973caa93c113731e9a305856aaadcb2cba6f0e592c45c"}
Jul 02 14:36:28.425577 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.425559    2551 scope.go:117] "RemoveContainer" containerID="79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31"
Jul 02 14:36:28.435148 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.435127    2551 scope.go:117] "RemoveContainer" containerID="8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3"
Jul 02 14:36:28.435421 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:36:28.435402    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3\": container with ID starting with 8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3 not found: ID does not exist" containerID="8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3"
Jul 02 14:36:28.435527 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.435434    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3"} err="failed to get container status \"8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3\": rpc error: code = NotFound desc = could not find container \"8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3\": container with ID starting with 8cc1668f8328c6c8eb6fc735fad5f79bf7b403db09ddd81a14088cdfc13d41e3 not found: ID does not exist"
Jul 02 14:36:28.435527 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.435457    2551 scope.go:117] "RemoveContainer" containerID="79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31"
Jul 02 14:36:28.435744 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:36:28.435727    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31\": container with ID starting with 79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31 not found: ID does not exist" containerID="79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31"
Jul 02 14:36:28.435783 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.435751    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31"} err="failed to get container status \"79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31\": rpc error: code = NotFound desc = could not find container \"79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31\": container with ID starting with 79aa552b4710f3719f9033d91197239517b94ec397aef8445e21d85a783ddd31 not found: ID does not exist"
Jul 02 14:36:28.435783 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.435769    2551 scope.go:117] "RemoveContainer" containerID="d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e"
Jul 02 14:36:28.442755 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.442732    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"]
Jul 02 14:36:28.443594 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.443566    2551 scope.go:117] "RemoveContainer" containerID="2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9"
Jul 02 14:36:28.447651 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.447628    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1c2b7-predictor-6f6dfd7649-vf2fc"]
Jul 02 14:36:28.451736 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.451718    2551 scope.go:117] "RemoveContainer" containerID="d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e"
Jul 02 14:36:28.451996 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:36:28.451981    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e\": container with ID starting with d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e not found: ID does not exist" containerID="d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e"
Jul 02 14:36:28.452040 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.452003    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e"} err="failed to get container status \"d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e\": rpc error: code = NotFound desc = could not find container \"d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e\": container with ID starting with d2932304eb07f56f333ed2f4219cb7e6a6bc1b30964c946fc34c52f9a4ae5a6e not found: ID does not exist"
Jul 02 14:36:28.452040 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.452027    2551 scope.go:117] "RemoveContainer" containerID="2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9"
Jul 02 14:36:28.452245 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:36:28.452229    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9\": container with ID starting with 2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9 not found: ID does not exist" containerID="2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9"
Jul 02 14:36:28.452281 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.452249    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9"} err="failed to get container status \"2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9\": rpc error: code = NotFound desc = could not find container \"2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9\": container with ID starting with 2605d8f03d0f08d66a19339fec4c90f56808fd80f918a6c5a47bf21ca0b354e9 not found: ID does not exist"
Jul 02 14:36:28.458824 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.458800    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"]
Jul 02 14:36:28.463657 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:28.463636    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1c2b7-predictor-768bc57fc6-xclr4"]
Jul 02 14:36:29.749375 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:29.749342    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" path="/var/lib/kubelet/pods/55158916-8aa3-4215-9c83-02b1224d1a62/volumes"
Jul 02 14:36:29.749771 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:29.749757    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" path="/var/lib/kubelet/pods/942cb8ee-ae7e-412b-a8ed-c2a5b3628019/volumes"
Jul 02 14:36:31.409242 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:31.409216    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:36:31.409667 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:31.409642    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 14:36:32.414807 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:32.414776    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:36:32.415398 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:32.415370    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 14:36:33.777533 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:33.777507    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:36:33.777961 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:33.777760    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:36:41.410086 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:41.410043    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 14:36:42.415988 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:42.415944    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 14:36:51.410235 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:51.410188    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 14:36:52.416152 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:36:52.416110    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 14:37:01.410444 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:01.410403    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 02 14:37:02.415776 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:02.415740    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 02 14:37:11.410191 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:11.410162    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:37:12.416677 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:12.416640    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:37:20.203044 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.203010    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"]
Jul 02 14:37:20.203572 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.203282    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container" containerID="cri-o://a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5" gracePeriod=30
Jul 02 14:37:20.203572 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.203375    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kube-rbac-proxy" containerID="cri-o://776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca" gracePeriod=30
Jul 02 14:37:20.223433 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.223408    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"]
Jul 02 14:37:20.223834 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.223803    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container"
Jul 02 14:37:20.223834 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.223828    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container"
Jul 02 14:37:20.223993 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.223844    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kube-rbac-proxy"
Jul 02 14:37:20.223993 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.223852    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kube-rbac-proxy"
Jul 02 14:37:20.223993 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.223865    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container"
Jul 02 14:37:20.223993 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.223873    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container"
Jul 02 14:37:20.223993 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.223907    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kube-rbac-proxy"
Jul 02 14:37:20.223993 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.223915    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kube-rbac-proxy"
Jul 02 14:37:20.224232 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.224021    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kube-rbac-proxy"
Jul 02 14:37:20.224232 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.224036    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kube-rbac-proxy"
Jul 02 14:37:20.224232 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.224047    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="55158916-8aa3-4215-9c83-02b1224d1a62" containerName="kserve-container"
Jul 02 14:37:20.224232 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.224058    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="942cb8ee-ae7e-412b-a8ed-c2a5b3628019" containerName="kserve-container"
Jul 02 14:37:20.232430 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.232403    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.235223 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.235195    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8937c-predictor-serving-cert\""
Jul 02 14:37:20.235338 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.235264    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8937c-kube-rbac-proxy-sar-config\""
Jul 02 14:37:20.238378 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.238358    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"]
Jul 02 14:37:20.321262 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.321225    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"]
Jul 02 14:37:20.325134 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.325115    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.327835 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.327812    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8937c-predictor-serving-cert\""
Jul 02 14:37:20.328219 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.328202    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8937c-kube-rbac-proxy-sar-config\""
Jul 02 14:37:20.336215 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.336192    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"]
Jul 02 14:37:20.339210 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.339184    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adc7e05a-8111-434b-8c37-8804fa0c10f4-proxy-tls\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.339310 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.339228    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-api-access-qcw82\") pod \"success-200-isvc-8937c-predictor-775f4ccc46-j956h\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") " pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.339359 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.339330    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8937c-predictor-775f4ccc46-j956h\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") " pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.339411 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.339383    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.339510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.339487    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrpc\" (UniqueName: \"kubernetes.io/projected/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-api-access-jfrpc\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.339599 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.339584    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130d2b67-9d70-4a56-b596-122ef94e67cf-proxy-tls\") pod \"success-200-isvc-8937c-predictor-775f4ccc46-j956h\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") " pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.400797 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.400761    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"]
Jul 02 14:37:20.401025 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.400997    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container" containerID="cri-o://7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e" gracePeriod=30
Jul 02 14:37:20.401146 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.401066    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kube-rbac-proxy" containerID="cri-o://ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8" gracePeriod=30
Jul 02 14:37:20.440518 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.440481    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130d2b67-9d70-4a56-b596-122ef94e67cf-proxy-tls\") pod \"success-200-isvc-8937c-predictor-775f4ccc46-j956h\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") " pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.440649 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.440545    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adc7e05a-8111-434b-8c37-8804fa0c10f4-proxy-tls\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.440649 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.440573    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-api-access-qcw82\") pod \"success-200-isvc-8937c-predictor-775f4ccc46-j956h\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") " pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.440649 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.440618    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8937c-predictor-775f4ccc46-j956h\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") " pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.440805 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.440657    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.440805 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:20.440682    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8937c-predictor-serving-cert: secret "error-404-isvc-8937c-predictor-serving-cert" not found
Jul 02 14:37:20.440805 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:20.440764    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adc7e05a-8111-434b-8c37-8804fa0c10f4-proxy-tls podName:adc7e05a-8111-434b-8c37-8804fa0c10f4 nodeName:}" failed. No retries permitted until 2026-07-02 14:37:20.94074453 +0000 UTC m=+1247.704129845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/adc7e05a-8111-434b-8c37-8804fa0c10f4-proxy-tls") pod "error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" (UID: "adc7e05a-8111-434b-8c37-8804fa0c10f4") : secret "error-404-isvc-8937c-predictor-serving-cert" not found
Jul 02 14:37:20.441023 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.440685    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrpc\" (UniqueName: \"kubernetes.io/projected/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-api-access-jfrpc\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.441358 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.441338    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8937c-predictor-775f4ccc46-j956h\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") " pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.441591 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.441568    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.443109 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.443087    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130d2b67-9d70-4a56-b596-122ef94e67cf-proxy-tls\") pod \"success-200-isvc-8937c-predictor-775f4ccc46-j956h\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") " pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.451995 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.451974    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-api-access-qcw82\") pod \"success-200-isvc-8937c-predictor-775f4ccc46-j956h\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") " pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.455574 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.455530    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrpc\" (UniqueName: \"kubernetes.io/projected/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-api-access-jfrpc\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.544021 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.543981    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:20.626411 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.626367    2551 generic.go:356] "Generic (PLEG): container finished" podID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerID="776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca" exitCode=2
Jul 02 14:37:20.626593 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.626442    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" event={"ID":"9f7a0abb-8b1f-4132-b57f-ba9437271dbd","Type":"ContainerDied","Data":"776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca"}
Jul 02 14:37:20.628457 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.628433    2551 generic.go:356] "Generic (PLEG): container finished" podID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerID="ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8" exitCode=2
Jul 02 14:37:20.628581 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.628500    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" event={"ID":"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c","Type":"ContainerDied","Data":"ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8"}
Jul 02 14:37:20.707330 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.707305    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"]
Jul 02 14:37:20.709787 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:37:20.709765    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130d2b67_9d70_4a56_b596_122ef94e67cf.slice/crio-e7e7cc18bc85e2f1fcd136709da6c0d7c0964ee5f1d44d66326255ed4c98272d WatchSource:0}: Error finding container e7e7cc18bc85e2f1fcd136709da6c0d7c0964ee5f1d44d66326255ed4c98272d: Status 404 returned error can't find the container with id e7e7cc18bc85e2f1fcd136709da6c0d7c0964ee5f1d44d66326255ed4c98272d
Jul 02 14:37:20.945942 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.945889    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adc7e05a-8111-434b-8c37-8804fa0c10f4-proxy-tls\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:20.948213 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:20.948193    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adc7e05a-8111-434b-8c37-8804fa0c10f4-proxy-tls\") pod \"error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") " pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:21.235605 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.235568    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:21.373993 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.373970    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"]
Jul 02 14:37:21.376031 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:37:21.376002    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc7e05a_8111_434b_8c37_8804fa0c10f4.slice/crio-844dfca2e5f959d6a7c7518c61925d22b084401ed733b8fb26902bfa08d47e0e WatchSource:0}: Error finding container 844dfca2e5f959d6a7c7518c61925d22b084401ed733b8fb26902bfa08d47e0e: Status 404 returned error can't find the container with id 844dfca2e5f959d6a7c7518c61925d22b084401ed733b8fb26902bfa08d47e0e
Jul 02 14:37:21.634520 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.634482    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" event={"ID":"adc7e05a-8111-434b-8c37-8804fa0c10f4","Type":"ContainerStarted","Data":"7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44"}
Jul 02 14:37:21.634714 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.634558    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" event={"ID":"adc7e05a-8111-434b-8c37-8804fa0c10f4","Type":"ContainerStarted","Data":"fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f"}
Jul 02 14:37:21.634714 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.634590    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" event={"ID":"adc7e05a-8111-434b-8c37-8804fa0c10f4","Type":"ContainerStarted","Data":"844dfca2e5f959d6a7c7518c61925d22b084401ed733b8fb26902bfa08d47e0e"}
Jul 02 14:37:21.634714 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.634620    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:21.636218 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.636195    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" event={"ID":"130d2b67-9d70-4a56-b596-122ef94e67cf","Type":"ContainerStarted","Data":"3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce"}
Jul 02 14:37:21.636335 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.636247    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" event={"ID":"130d2b67-9d70-4a56-b596-122ef94e67cf","Type":"ContainerStarted","Data":"9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2"}
Jul 02 14:37:21.636335 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.636282    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:21.636335 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.636311    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" event={"ID":"130d2b67-9d70-4a56-b596-122ef94e67cf","Type":"ContainerStarted","Data":"e7e7cc18bc85e2f1fcd136709da6c0d7c0964ee5f1d44d66326255ed4c98272d"}
Jul 02 14:37:21.653869 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.653826    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podStartSLOduration=1.6538118050000001 podStartE2EDuration="1.653811805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:37:21.651590422 +0000 UTC m=+1248.414975743" watchObservedRunningTime="2026-07-02 14:37:21.653811805 +0000 UTC m=+1248.417197127"
Jul 02 14:37:21.668340 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:21.668292    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podStartSLOduration=1.668273984 podStartE2EDuration="1.668273984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:37:21.667497532 +0000 UTC m=+1248.430882854" watchObservedRunningTime="2026-07-02 14:37:21.668273984 +0000 UTC m=+1248.431659306"
Jul 02 14:37:22.458380 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:22.458339    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 02 14:37:22.463758 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:22.463727    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 02 14:37:22.640665 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:22.640632    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:22.640849 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:22.640685    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:22.641647 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:22.641618    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 14:37:22.641647 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:22.641631    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 14:37:23.463044 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:23.463004    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 02 14:37:23.467460 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:23.467431    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 02 14:37:23.644806 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:23.644771    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 14:37:23.644954 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:23.644843    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 14:37:24.257881 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.257856    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:37:24.348512 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.348436    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:37:24.373423 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.373398    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmtsv\" (UniqueName: \"kubernetes.io/projected/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-api-access-qmtsv\") pod \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") "
Jul 02 14:37:24.373562 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.373444    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-proxy-tls\") pod \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") "
Jul 02 14:37:24.373562 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.373487    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-rbac-proxy-sar-config\") pod \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") "
Jul 02 14:37:24.373562 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.373546    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-proxy-tls\") pod \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") "
Jul 02 14:37:24.373727 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.373596    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-rbac-proxy-sar-config\") pod \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\" (UID: \"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c\") "
Jul 02 14:37:24.373727 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.373634    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xwpt\" (UniqueName: \"kubernetes.io/projected/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-api-access-2xwpt\") pod \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\" (UID: \"9f7a0abb-8b1f-4132-b57f-ba9437271dbd\") "
Jul 02 14:37:24.373921 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.373877    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f7a0abb-8b1f-4132-b57f-ba9437271dbd" (UID: "9f7a0abb-8b1f-4132-b57f-ba9437271dbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:37:24.374048 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.373935    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" (UID: "7d71ebfa-3c34-40a6-9373-36a7a7a9a19c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:37:24.375693 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.375667    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-api-access-2xwpt" (OuterVolumeSpecName: "kube-api-access-2xwpt") pod "9f7a0abb-8b1f-4132-b57f-ba9437271dbd" (UID: "9f7a0abb-8b1f-4132-b57f-ba9437271dbd"). InnerVolumeSpecName "kube-api-access-2xwpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:37:24.375776 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.375696    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-api-access-qmtsv" (OuterVolumeSpecName: "kube-api-access-qmtsv") pod "7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" (UID: "7d71ebfa-3c34-40a6-9373-36a7a7a9a19c"). InnerVolumeSpecName "kube-api-access-qmtsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:37:24.375776 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.375713    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" (UID: "7d71ebfa-3c34-40a6-9373-36a7a7a9a19c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:37:24.376123 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.376110    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f7a0abb-8b1f-4132-b57f-ba9437271dbd" (UID: "9f7a0abb-8b1f-4132-b57f-ba9437271dbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:37:24.475097 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.475069    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmtsv\" (UniqueName: \"kubernetes.io/projected/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-api-access-qmtsv\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:24.475097 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.475095    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:24.475097 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.475105    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:24.475533 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.475113    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:24.475533 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.475123    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:24.475533 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.475130    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwpt\" (UniqueName: \"kubernetes.io/projected/9f7a0abb-8b1f-4132-b57f-ba9437271dbd-kube-api-access-2xwpt\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:24.650431 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.650352    2551 generic.go:356] "Generic (PLEG): container finished" podID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerID="a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5" exitCode=0
Jul 02 14:37:24.650431 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.650400    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"
Jul 02 14:37:24.650765 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.650436    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" event={"ID":"9f7a0abb-8b1f-4132-b57f-ba9437271dbd","Type":"ContainerDied","Data":"a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5"}
Jul 02 14:37:24.650765 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.650553    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh" event={"ID":"9f7a0abb-8b1f-4132-b57f-ba9437271dbd","Type":"ContainerDied","Data":"e20ccc4d1d593bbd34b80e9d096a9fd263d8795f5a759d158796c48a2d86111f"}
Jul 02 14:37:24.650765 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.650592    2551 scope.go:117] "RemoveContainer" containerID="776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca"
Jul 02 14:37:24.652356 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.652337    2551 generic.go:356] "Generic (PLEG): container finished" podID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerID="7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e" exitCode=0
Jul 02 14:37:24.652456 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.652402    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"
Jul 02 14:37:24.652456 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.652412    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" event={"ID":"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c","Type":"ContainerDied","Data":"7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e"}
Jul 02 14:37:24.652571 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.652508    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s" event={"ID":"7d71ebfa-3c34-40a6-9373-36a7a7a9a19c","Type":"ContainerDied","Data":"0e3b8ebed6e664c4bea1de8c1614c5fadfc461cb67e765e7c32a3bcbf5bce211"}
Jul 02 14:37:24.660877 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.660864    2551 scope.go:117] "RemoveContainer" containerID="a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5"
Jul 02 14:37:24.668110 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.668095    2551 scope.go:117] "RemoveContainer" containerID="776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca"
Jul 02 14:37:24.668351 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:24.668333    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca\": container with ID starting with 776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca not found: ID does not exist" containerID="776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca"
Jul 02 14:37:24.668404 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.668360    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca"} err="failed to get container status \"776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca\": rpc error: code = NotFound desc = could not find container \"776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca\": container with ID starting with 776d97735a1aa2a32aac604c49af935fbf025156929f5106a4f1ad65f58542ca not found: ID does not exist"
Jul 02 14:37:24.668404 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.668376    2551 scope.go:117] "RemoveContainer" containerID="a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5"
Jul 02 14:37:24.668628 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:24.668610    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5\": container with ID starting with a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5 not found: ID does not exist" containerID="a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5"
Jul 02 14:37:24.668669 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.668633    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5"} err="failed to get container status \"a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5\": rpc error: code = NotFound desc = could not find container \"a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5\": container with ID starting with a173ebeb195db8cf3de48cd4eaed764d653ed441321bd58f1957fdecb05a60a5 not found: ID does not exist"
Jul 02 14:37:24.668669 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.668649    2551 scope.go:117] "RemoveContainer" containerID="ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8"
Jul 02 14:37:24.675739 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.675717    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"]
Jul 02 14:37:24.675916 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.675901    2551 scope.go:117] "RemoveContainer" containerID="7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e"
Jul 02 14:37:24.680047 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.680026    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9a7ca-predictor-78f5cf9c68-56vsh"]
Jul 02 14:37:24.683324 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.683311    2551 scope.go:117] "RemoveContainer" containerID="ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8"
Jul 02 14:37:24.683586 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:24.683570    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8\": container with ID starting with ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8 not found: ID does not exist" containerID="ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8"
Jul 02 14:37:24.683632 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.683592    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8"} err="failed to get container status \"ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8\": rpc error: code = NotFound desc = could not find container \"ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8\": container with ID starting with ccb2dfe1a95191016420a53c8805d64581d1297a8732662f01e90959e0f392b8 not found: ID does not exist"
Jul 02 14:37:24.683632 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.683608    2551 scope.go:117] "RemoveContainer" containerID="7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e"
Jul 02 14:37:24.683839 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:24.683822    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e\": container with ID starting with 7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e not found: ID does not exist" containerID="7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e"
Jul 02 14:37:24.683872 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.683846    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e"} err="failed to get container status \"7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e\": rpc error: code = NotFound desc = could not find container \"7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e\": container with ID starting with 7a5bc010ceb72f8cbc7724b609500b1a53e86d65bf53d4c926e1d4e1f727351e not found: ID does not exist"
Jul 02 14:37:24.692009 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.691989    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"]
Jul 02 14:37:24.695058 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:24.695034    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a7ca-predictor-c544cc7fc-jks4s"]
Jul 02 14:37:25.749207 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:25.749174    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" path="/var/lib/kubelet/pods/7d71ebfa-3c34-40a6-9373-36a7a7a9a19c/volumes"
Jul 02 14:37:25.749609 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:25.749595    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" path="/var/lib/kubelet/pods/9f7a0abb-8b1f-4132-b57f-ba9437271dbd/volumes"
Jul 02 14:37:28.648834 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:28.648808    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:37:28.649299 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:28.649137    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:37:28.649299 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:28.649245    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 14:37:28.649628 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:28.649602    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 14:37:34.694370 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.694331    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"]
Jul 02 14:37:34.694888 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.694688    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container" containerID="cri-o://6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a" gracePeriod=30
Jul 02 14:37:34.695097 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.695069    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kube-rbac-proxy" containerID="cri-o://d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855" gracePeriod=30
Jul 02 14:37:34.718819 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.718792    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"]
Jul 02 14:37:34.719127 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719116    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kube-rbac-proxy"
Jul 02 14:37:34.719174 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719129    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kube-rbac-proxy"
Jul 02 14:37:34.719174 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719141    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container"
Jul 02 14:37:34.719174 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719146    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container"
Jul 02 14:37:34.719174 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719161    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container"
Jul 02 14:37:34.719174 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719166    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container"
Jul 02 14:37:34.719322 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719179    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kube-rbac-proxy"
Jul 02 14:37:34.719322 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719184    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kube-rbac-proxy"
Jul 02 14:37:34.719322 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719234    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kube-rbac-proxy"
Jul 02 14:37:34.719322 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719242    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kserve-container"
Jul 02 14:37:34.719322 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719248    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7a0abb-8b1f-4132-b57f-ba9437271dbd" containerName="kserve-container"
Jul 02 14:37:34.719322 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.719254    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d71ebfa-3c34-40a6-9373-36a7a7a9a19c" containerName="kube-rbac-proxy"
Jul 02 14:37:34.723972 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.723953    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:34.726740 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.726719    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-57d01-kube-rbac-proxy-sar-config\""
Jul 02 14:37:34.726845 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.726719    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-57d01-predictor-serving-cert\""
Jul 02 14:37:34.732099 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.732078    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"]
Jul 02 14:37:34.818372 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.818346    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"]
Jul 02 14:37:34.821816 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.821796    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:34.824392 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.824374    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-57d01-predictor-serving-cert\""
Jul 02 14:37:34.824508 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.824376    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-57d01-kube-rbac-proxy-sar-config\""
Jul 02 14:37:34.831043 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.831021    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"]
Jul 02 14:37:34.861753 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.861726    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c2fe9bf-8edd-416f-b699-67c932039ec6-proxy-tls\") pod \"error-404-isvc-57d01-predictor-7b7d9b5777-88mnx\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") " pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:34.861901 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.861772    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b829695-5466-4067-a183-a1a7434cb2c8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:34.861901 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.861799    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-57d01-predictor-7b7d9b5777-88mnx\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") " pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:34.861981 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.861944    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw4f2\" (UniqueName: \"kubernetes.io/projected/9b829695-5466-4067-a183-a1a7434cb2c8-kube-api-access-lw4f2\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:34.862022 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.861986    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b829695-5466-4067-a183-a1a7434cb2c8-proxy-tls\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:34.862060 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.862018    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s567n\" (UniqueName: \"kubernetes.io/projected/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-api-access-s567n\") pod \"error-404-isvc-57d01-predictor-7b7d9b5777-88mnx\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") " pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:34.865779 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.865752    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"]
Jul 02 14:37:34.866085 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.866058    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container" containerID="cri-o://a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006" gracePeriod=30
Jul 02 14:37:34.866180 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.866089    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kube-rbac-proxy" containerID="cri-o://3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2" gracePeriod=30
Jul 02 14:37:34.963441 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.963360    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b829695-5466-4067-a183-a1a7434cb2c8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:34.963441 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.963398    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-57d01-predictor-7b7d9b5777-88mnx\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") " pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:34.963682 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.963464    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw4f2\" (UniqueName: \"kubernetes.io/projected/9b829695-5466-4067-a183-a1a7434cb2c8-kube-api-access-lw4f2\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:34.963682 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.963511    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b829695-5466-4067-a183-a1a7434cb2c8-proxy-tls\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:34.963682 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.963549    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s567n\" (UniqueName: \"kubernetes.io/projected/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-api-access-s567n\") pod \"error-404-isvc-57d01-predictor-7b7d9b5777-88mnx\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") " pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:34.963682 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.963587    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c2fe9bf-8edd-416f-b699-67c932039ec6-proxy-tls\") pod \"error-404-isvc-57d01-predictor-7b7d9b5777-88mnx\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") " pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:34.963872 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:34.963800    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-57d01-predictor-serving-cert: secret "success-200-isvc-57d01-predictor-serving-cert" not found
Jul 02 14:37:34.963914 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:34.963878    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b829695-5466-4067-a183-a1a7434cb2c8-proxy-tls podName:9b829695-5466-4067-a183-a1a7434cb2c8 nodeName:}" failed. No retries permitted until 2026-07-02 14:37:35.463858088 +0000 UTC m=+1262.227243399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b829695-5466-4067-a183-a1a7434cb2c8-proxy-tls") pod "success-200-isvc-57d01-predictor-5bb7475544-nw2mc" (UID: "9b829695-5466-4067-a183-a1a7434cb2c8") : secret "success-200-isvc-57d01-predictor-serving-cert" not found
Jul 02 14:37:34.964205 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.964178    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b829695-5466-4067-a183-a1a7434cb2c8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:34.964205 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.964202    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-57d01-predictor-7b7d9b5777-88mnx\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") " pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:34.966094 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.966076    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c2fe9bf-8edd-416f-b699-67c932039ec6-proxy-tls\") pod \"error-404-isvc-57d01-predictor-7b7d9b5777-88mnx\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") " pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:34.975652 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.975629    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw4f2\" (UniqueName: \"kubernetes.io/projected/9b829695-5466-4067-a183-a1a7434cb2c8-kube-api-access-lw4f2\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:34.976148 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:34.976128    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s567n\" (UniqueName: \"kubernetes.io/projected/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-api-access-s567n\") pod \"error-404-isvc-57d01-predictor-7b7d9b5777-88mnx\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") " pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:35.134601 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.134567    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:35.264809 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.264648    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"]
Jul 02 14:37:35.266843 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:37:35.266818    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2fe9bf_8edd_416f_b699_67c932039ec6.slice/crio-3422bfa1f1d02bc155cebd0dcf2c37228323f66ea0b1465346782a5a1f9d022b WatchSource:0}: Error finding container 3422bfa1f1d02bc155cebd0dcf2c37228323f66ea0b1465346782a5a1f9d022b: Status 404 returned error can't find the container with id 3422bfa1f1d02bc155cebd0dcf2c37228323f66ea0b1465346782a5a1f9d022b
Jul 02 14:37:35.468262 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.468184    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b829695-5466-4067-a183-a1a7434cb2c8-proxy-tls\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:35.470641 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.470613    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b829695-5466-4067-a183-a1a7434cb2c8-proxy-tls\") pod \"success-200-isvc-57d01-predictor-5bb7475544-nw2mc\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") " pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:35.635416 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.635377    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:35.703745 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.703709    2551 generic.go:356] "Generic (PLEG): container finished" podID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerID="d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855" exitCode=2
Jul 02 14:37:35.704109 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.703779    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" event={"ID":"3a0e9c74-f5b3-407c-a5d5-15377a551448","Type":"ContainerDied","Data":"d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855"}
Jul 02 14:37:35.705755 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.705733    2551 generic.go:356] "Generic (PLEG): container finished" podID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerID="3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2" exitCode=2
Jul 02 14:37:35.705879 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.705812    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" event={"ID":"ae07db8f-0d68-470b-afaa-b5cee9315f7e","Type":"ContainerDied","Data":"3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2"}
Jul 02 14:37:35.707663 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.707638    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" event={"ID":"8c2fe9bf-8edd-416f-b699-67c932039ec6","Type":"ContainerStarted","Data":"35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d"}
Jul 02 14:37:35.707757 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.707692    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" event={"ID":"8c2fe9bf-8edd-416f-b699-67c932039ec6","Type":"ContainerStarted","Data":"261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812"}
Jul 02 14:37:35.707757 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.707724    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" event={"ID":"8c2fe9bf-8edd-416f-b699-67c932039ec6","Type":"ContainerStarted","Data":"3422bfa1f1d02bc155cebd0dcf2c37228323f66ea0b1465346782a5a1f9d022b"}
Jul 02 14:37:35.707835 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.707758    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:35.727284 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.727233    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podStartSLOduration=1.727213974 podStartE2EDuration="1.727213974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:37:35.726202007 +0000 UTC m=+1262.489587342" watchObservedRunningTime="2026-07-02 14:37:35.727213974 +0000 UTC m=+1262.490599297"
Jul 02 14:37:35.771211 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:35.771188    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"]
Jul 02 14:37:35.773775 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:37:35.773752    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b829695_5466_4067_a183_a1a7434cb2c8.slice/crio-585795edae166a343243fc10c4187a466819c8a5ef5f285ba7c35f4fb3e5cf91 WatchSource:0}: Error finding container 585795edae166a343243fc10c4187a466819c8a5ef5f285ba7c35f4fb3e5cf91: Status 404 returned error can't find the container with id 585795edae166a343243fc10c4187a466819c8a5ef5f285ba7c35f4fb3e5cf91
Jul 02 14:37:36.406142 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.406095    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 02 14:37:36.713927 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.713812    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" event={"ID":"9b829695-5466-4067-a183-a1a7434cb2c8","Type":"ContainerStarted","Data":"c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e"}
Jul 02 14:37:36.713927 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.713901    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:36.714372 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.713933    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" event={"ID":"9b829695-5466-4067-a183-a1a7434cb2c8","Type":"ContainerStarted","Data":"b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567"}
Jul 02 14:37:36.714372 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.713968    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:36.714372 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.713993    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" event={"ID":"9b829695-5466-4067-a183-a1a7434cb2c8","Type":"ContainerStarted","Data":"585795edae166a343243fc10c4187a466819c8a5ef5f285ba7c35f4fb3e5cf91"}
Jul 02 14:37:36.714372 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.714193    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:36.715747 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.715724    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 14:37:36.715794 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.715743    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:37:36.733736 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:36.733687    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podStartSLOduration=2.733671433 podStartE2EDuration="2.733671433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:37:36.731621339 +0000 UTC m=+1263.495006661" watchObservedRunningTime="2026-07-02 14:37:36.733671433 +0000 UTC m=+1263.497056754"
Jul 02 14:37:37.410890 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:37.410848    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 02 14:37:37.718598 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:37.718501    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:37:37.718598 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:37.718522    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 14:37:38.334920 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.334896    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:37:38.337530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.337510    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:37:38.396783 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.396697    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae07db8f-0d68-470b-afaa-b5cee9315f7e-proxy-tls\") pod \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") "
Jul 02 14:37:38.396783 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.396735    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0e9c74-f5b3-407c-a5d5-15377a551448-proxy-tls\") pod \"3a0e9c74-f5b3-407c-a5d5-15377a551448\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") "
Jul 02 14:37:38.396783 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.396768    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-rbac-proxy-sar-config\") pod \"3a0e9c74-f5b3-407c-a5d5-15377a551448\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") "
Jul 02 14:37:38.397023 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.396841    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82hvn\" (UniqueName: \"kubernetes.io/projected/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-api-access-82hvn\") pod \"3a0e9c74-f5b3-407c-a5d5-15377a551448\" (UID: \"3a0e9c74-f5b3-407c-a5d5-15377a551448\") "
Jul 02 14:37:38.397084 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.397030    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-rbac-proxy-sar-config\") pod \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") "
Jul 02 14:37:38.397130 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.397080    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prx2s\" (UniqueName: \"kubernetes.io/projected/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-api-access-prx2s\") pod \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\" (UID: \"ae07db8f-0d68-470b-afaa-b5cee9315f7e\") "
Jul 02 14:37:38.397343 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.397309    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a0e9c74-f5b3-407c-a5d5-15377a551448" (UID: "3a0e9c74-f5b3-407c-a5d5-15377a551448"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:37:38.397343 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.397327    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae07db8f-0d68-470b-afaa-b5cee9315f7e" (UID: "ae07db8f-0d68-470b-afaa-b5cee9315f7e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:37:38.398907 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.398878    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae07db8f-0d68-470b-afaa-b5cee9315f7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae07db8f-0d68-470b-afaa-b5cee9315f7e" (UID: "ae07db8f-0d68-470b-afaa-b5cee9315f7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:37:38.399092 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.399076    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-api-access-prx2s" (OuterVolumeSpecName: "kube-api-access-prx2s") pod "ae07db8f-0d68-470b-afaa-b5cee9315f7e" (UID: "ae07db8f-0d68-470b-afaa-b5cee9315f7e"). InnerVolumeSpecName "kube-api-access-prx2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:37:38.399298 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.399272    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0e9c74-f5b3-407c-a5d5-15377a551448-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a0e9c74-f5b3-407c-a5d5-15377a551448" (UID: "3a0e9c74-f5b3-407c-a5d5-15377a551448"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:37:38.399298 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.399282    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-api-access-82hvn" (OuterVolumeSpecName: "kube-api-access-82hvn") pod "3a0e9c74-f5b3-407c-a5d5-15377a551448" (UID: "3a0e9c74-f5b3-407c-a5d5-15377a551448"). InnerVolumeSpecName "kube-api-access-82hvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:37:38.497943 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.497905    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hvn\" (UniqueName: \"kubernetes.io/projected/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-api-access-82hvn\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:38.497943 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.497939    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:38.497943 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.497949    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prx2s\" (UniqueName: \"kubernetes.io/projected/ae07db8f-0d68-470b-afaa-b5cee9315f7e-kube-api-access-prx2s\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:38.498160 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.497960    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae07db8f-0d68-470b-afaa-b5cee9315f7e-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:38.498160 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.497970    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a0e9c74-f5b3-407c-a5d5-15377a551448-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:38.498160 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.497978    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a0e9c74-f5b3-407c-a5d5-15377a551448-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:37:38.649991 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.649894    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 14:37:38.650147 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.649897    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 14:37:38.724020 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.723977    2551 generic.go:356] "Generic (PLEG): container finished" podID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerID="6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a" exitCode=0
Jul 02 14:37:38.724497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.724038    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"
Jul 02 14:37:38.724497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.724032    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" event={"ID":"3a0e9c74-f5b3-407c-a5d5-15377a551448","Type":"ContainerDied","Data":"6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a"}
Jul 02 14:37:38.724497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.724231    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7" event={"ID":"3a0e9c74-f5b3-407c-a5d5-15377a551448","Type":"ContainerDied","Data":"5d8f5c47f308dd9f8458567d531769fbea30535a8a31197d63a16b09aabc9e8d"}
Jul 02 14:37:38.724497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.724272    2551 scope.go:117] "RemoveContainer" containerID="d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855"
Jul 02 14:37:38.725843 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.725818    2551 generic.go:356] "Generic (PLEG): container finished" podID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerID="a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006" exitCode=0
Jul 02 14:37:38.725970 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.725862    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"
Jul 02 14:37:38.725970 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.725858    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" event={"ID":"ae07db8f-0d68-470b-afaa-b5cee9315f7e","Type":"ContainerDied","Data":"a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006"}
Jul 02 14:37:38.726090 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.725980    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m" event={"ID":"ae07db8f-0d68-470b-afaa-b5cee9315f7e","Type":"ContainerDied","Data":"387865ba6e7959fe9c81409440266379f3c42156fdfda1c41c37f7a399f1ba10"}
Jul 02 14:37:38.733201 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.733166    2551 scope.go:117] "RemoveContainer" containerID="6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a"
Jul 02 14:37:38.740883 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.740864    2551 scope.go:117] "RemoveContainer" containerID="d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855"
Jul 02 14:37:38.741168 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:38.741146    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855\": container with ID starting with d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855 not found: ID does not exist" containerID="d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855"
Jul 02 14:37:38.741234 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.741174    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855"} err="failed to get container status \"d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855\": rpc error: code = NotFound desc = could not find container \"d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855\": container with ID starting with d91c49852f584a3699af9c957101e12953c3806f6ff080256ca20d1b30423855 not found: ID does not exist"
Jul 02 14:37:38.741234 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.741190    2551 scope.go:117] "RemoveContainer" containerID="6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a"
Jul 02 14:37:38.741416 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:38.741398    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a\": container with ID starting with 6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a not found: ID does not exist" containerID="6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a"
Jul 02 14:37:38.741456 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.741421    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a"} err="failed to get container status \"6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a\": rpc error: code = NotFound desc = could not find container \"6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a\": container with ID starting with 6cd2d069af2d1e1f4b57907c9b4cd2279e72d7f66af2640b2c1c3d3adc9a4e6a not found: ID does not exist"
Jul 02 14:37:38.741456 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.741437    2551 scope.go:117] "RemoveContainer" containerID="3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2"
Jul 02 14:37:38.748544 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.748523    2551 scope.go:117] "RemoveContainer" containerID="a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006"
Jul 02 14:37:38.750356 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.750330    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"]
Jul 02 14:37:38.754508 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.754487    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fbc8b-predictor-5f496dcc84-gpcr7"]
Jul 02 14:37:38.756221 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.756205    2551 scope.go:117] "RemoveContainer" containerID="3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2"
Jul 02 14:37:38.756494 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:38.756462    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2\": container with ID starting with 3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2 not found: ID does not exist" containerID="3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2"
Jul 02 14:37:38.756573 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.756506    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2"} err="failed to get container status \"3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2\": rpc error: code = NotFound desc = could not find container \"3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2\": container with ID starting with 3ec305b76c3963c3c0e18e87598b9e64df2ca2fbe5d240c29e40902bed765de2 not found: ID does not exist"
Jul 02 14:37:38.756573 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.756530    2551 scope.go:117] "RemoveContainer" containerID="a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006"
Jul 02 14:37:38.756769 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:37:38.756751    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006\": container with ID starting with a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006 not found: ID does not exist" containerID="a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006"
Jul 02 14:37:38.756808 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.756777    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006"} err="failed to get container status \"a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006\": rpc error: code = NotFound desc = could not find container \"a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006\": container with ID starting with a1ea4b92c204925ea1afd8512834056e15b777a2c7140f33f15b9e45046a4006 not found: ID does not exist"
Jul 02 14:37:38.767066 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.767041    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"]
Jul 02 14:37:38.771308 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:38.771289    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fbc8b-predictor-64c779dbb5-7br8m"]
Jul 02 14:37:39.748012 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:39.747984    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" path="/var/lib/kubelet/pods/3a0e9c74-f5b3-407c-a5d5-15377a551448/volumes"
Jul 02 14:37:39.748409 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:39.748396    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" path="/var/lib/kubelet/pods/ae07db8f-0d68-470b-afaa-b5cee9315f7e/volumes"
Jul 02 14:37:42.722231 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:42.722200    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:37:42.722680 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:42.722534    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:37:42.722680 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:42.722648    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:37:42.722968 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:42.722947    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 14:37:48.649285 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:48.649242    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 14:37:48.649710 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:48.649552    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 14:37:52.723297 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:52.723256    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 14:37:52.723698 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:52.723256    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:37:58.649411 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:58.649368    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 14:37:58.649798 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:37:58.649517    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 14:38:02.723241 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:02.723204    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 14:38:02.723624 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:02.723208    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:38:08.649288 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:08.649247    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 14:38:08.649856 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:08.649560    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 02 14:38:12.723155 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:12.723119    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:38:12.723544 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:12.723127    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 14:38:18.650603 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:18.650574    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:38:18.650990 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:18.650654    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:38:22.723378 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:22.723337    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:38:22.723947 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:22.723580    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:38:32.724012 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:32.723982    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:38:44.579663 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.579628    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"]
Jul 02 14:38:44.580223 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.579901    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container" containerID="cri-o://fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f" gracePeriod=30
Jul 02 14:38:44.580223 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.579958    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kube-rbac-proxy" containerID="cri-o://7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44" gracePeriod=30
Jul 02 14:38:44.629838 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.629801    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"]
Jul 02 14:38:44.630135 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630123    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kube-rbac-proxy"
Jul 02 14:38:44.630135 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630136    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kube-rbac-proxy"
Jul 02 14:38:44.630242 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630149    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container"
Jul 02 14:38:44.630242 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630154    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container"
Jul 02 14:38:44.630242 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630162    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kube-rbac-proxy"
Jul 02 14:38:44.630242 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630168    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kube-rbac-proxy"
Jul 02 14:38:44.630242 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630180    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container"
Jul 02 14:38:44.630242 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630185    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container"
Jul 02 14:38:44.630242 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630232    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kserve-container"
Jul 02 14:38:44.630242 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630241    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kube-rbac-proxy"
Jul 02 14:38:44.630623 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630251    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae07db8f-0d68-470b-afaa-b5cee9315f7e" containerName="kserve-container"
Jul 02 14:38:44.630623 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.630256    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a0e9c74-f5b3-407c-a5d5-15377a551448" containerName="kube-rbac-proxy"
Jul 02 14:38:44.634738 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.634715    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:44.639283 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.639263    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c3f32-kube-rbac-proxy-sar-config\""
Jul 02 14:38:44.639790 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.639776    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c3f32-predictor-serving-cert\""
Jul 02 14:38:44.646430 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.646403    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"]
Jul 02 14:38:44.709520 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.709448    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"]
Jul 02 14:38:44.709829 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.709797    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container" containerID="cri-o://9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2" gracePeriod=30
Jul 02 14:38:44.710074 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.709894    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kube-rbac-proxy" containerID="cri-o://3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce" gracePeriod=30
Jul 02 14:38:44.713112 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.713089    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee64839-23c4-465a-a0c5-e6daf981a63c-proxy-tls\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:44.713301 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.713286    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:44.713440 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.713426    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5kcf\" (UniqueName: \"kubernetes.io/projected/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-api-access-r5kcf\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:44.755992 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.755960    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"]
Jul 02 14:38:44.759946 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.759926    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:44.762660 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.762621    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3f32-predictor-serving-cert\""
Jul 02 14:38:44.762873 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.762849    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3f32-kube-rbac-proxy-sar-config\""
Jul 02 14:38:44.771770 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.771740    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"]
Jul 02 14:38:44.814686 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.814650    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-448z4\" (UniqueName: \"kubernetes.io/projected/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-api-access-448z4\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:44.814857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.814697    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:44.814857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.814744    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:44.814857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.814807    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5kcf\" (UniqueName: \"kubernetes.io/projected/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-api-access-r5kcf\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:44.815030 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.814993    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee64839-23c4-465a-a0c5-e6daf981a63c-proxy-tls\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:44.815089 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.815030    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b869f464-c7f0-4c3a-9a25-29730c80d07b-proxy-tls\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:44.815159 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:38:44.815139    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-serving-cert: secret "success-200-isvc-c3f32-predictor-serving-cert" not found
Jul 02 14:38:44.815229 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:38:44.815217    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee64839-23c4-465a-a0c5-e6daf981a63c-proxy-tls podName:9ee64839-23c4-465a-a0c5-e6daf981a63c nodeName:}" failed. No retries permitted until 2026-07-02 14:38:45.315194794 +0000 UTC m=+1332.078580107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9ee64839-23c4-465a-a0c5-e6daf981a63c-proxy-tls") pod "success-200-isvc-c3f32-predictor-8449d84b64-z62pq" (UID: "9ee64839-23c4-465a-a0c5-e6daf981a63c") : secret "success-200-isvc-c3f32-predictor-serving-cert" not found
Jul 02 14:38:44.815509 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.815463    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:44.824525 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.824502    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5kcf\" (UniqueName: \"kubernetes.io/projected/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-api-access-r5kcf\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:44.915627 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.915595    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b869f464-c7f0-4c3a-9a25-29730c80d07b-proxy-tls\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:44.915827 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.915638    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-448z4\" (UniqueName: \"kubernetes.io/projected/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-api-access-448z4\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:44.915827 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.915668    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:44.915827 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:38:44.915751    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-serving-cert: secret "error-404-isvc-c3f32-predictor-serving-cert" not found
Jul 02 14:38:44.915827 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:38:44.915829    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b869f464-c7f0-4c3a-9a25-29730c80d07b-proxy-tls podName:b869f464-c7f0-4c3a-9a25-29730c80d07b nodeName:}" failed. No retries permitted until 2026-07-02 14:38:45.415802682 +0000 UTC m=+1332.179187988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b869f464-c7f0-4c3a-9a25-29730c80d07b-proxy-tls") pod "error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" (UID: "b869f464-c7f0-4c3a-9a25-29730c80d07b") : secret "error-404-isvc-c3f32-predictor-serving-cert" not found
Jul 02 14:38:44.916588 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.916567    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:44.927262 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:44.927222    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-448z4\" (UniqueName: \"kubernetes.io/projected/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-api-access-448z4\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:45.000254 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.000220    2551 generic.go:356] "Generic (PLEG): container finished" podID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerID="7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44" exitCode=2
Jul 02 14:38:45.000442 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.000291    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" event={"ID":"adc7e05a-8111-434b-8c37-8804fa0c10f4","Type":"ContainerDied","Data":"7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44"}
Jul 02 14:38:45.002182 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.002161    2551 generic.go:356] "Generic (PLEG): container finished" podID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerID="3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce" exitCode=2
Jul 02 14:38:45.002291 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.002185    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" event={"ID":"130d2b67-9d70-4a56-b596-122ef94e67cf","Type":"ContainerDied","Data":"3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce"}
Jul 02 14:38:45.318783 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.318749    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee64839-23c4-465a-a0c5-e6daf981a63c-proxy-tls\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:45.321125 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.321103    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee64839-23c4-465a-a0c5-e6daf981a63c-proxy-tls\") pod \"success-200-isvc-c3f32-predictor-8449d84b64-z62pq\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") " pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:45.420028 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.419989    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b869f464-c7f0-4c3a-9a25-29730c80d07b-proxy-tls\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:45.422336 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.422318    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b869f464-c7f0-4c3a-9a25-29730c80d07b-proxy-tls\") pod \"error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") " pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:45.546312 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.546277    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:45.674637 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.674602    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:45.679262 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.679235    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"]
Jul 02 14:38:45.683340 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:38:45.683315    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee64839_23c4_465a_a0c5_e6daf981a63c.slice/crio-f5d55619c2adb00623d452b6fd25fb6a5a7fbfeecabda80a7a40101238090adb WatchSource:0}: Error finding container f5d55619c2adb00623d452b6fd25fb6a5a7fbfeecabda80a7a40101238090adb: Status 404 returned error can't find the container with id f5d55619c2adb00623d452b6fd25fb6a5a7fbfeecabda80a7a40101238090adb
Jul 02 14:38:45.829123 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:45.829097    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"]
Jul 02 14:38:45.832924 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:38:45.832902    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb869f464_c7f0_4c3a_9a25_29730c80d07b.slice/crio-2e8074de72b1ce5e8de635a1f76399aa77c25182e5c810251e110f4e466e4213 WatchSource:0}: Error finding container 2e8074de72b1ce5e8de635a1f76399aa77c25182e5c810251e110f4e466e4213: Status 404 returned error can't find the container with id 2e8074de72b1ce5e8de635a1f76399aa77c25182e5c810251e110f4e466e4213
Jul 02 14:38:46.008190 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.008153    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" event={"ID":"9ee64839-23c4-465a-a0c5-e6daf981a63c","Type":"ContainerStarted","Data":"9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74"}
Jul 02 14:38:46.008355 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.008228    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" event={"ID":"9ee64839-23c4-465a-a0c5-e6daf981a63c","Type":"ContainerStarted","Data":"300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7"}
Jul 02 14:38:46.008355 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.008271    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:46.008355 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.008305    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:46.008355 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.008339    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" event={"ID":"9ee64839-23c4-465a-a0c5-e6daf981a63c","Type":"ContainerStarted","Data":"f5d55619c2adb00623d452b6fd25fb6a5a7fbfeecabda80a7a40101238090adb"}
Jul 02 14:38:46.009597 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.009566    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:38:46.010010 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.009988    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" event={"ID":"b869f464-c7f0-4c3a-9a25-29730c80d07b","Type":"ContainerStarted","Data":"7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408"}
Jul 02 14:38:46.010099 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.010037    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" event={"ID":"b869f464-c7f0-4c3a-9a25-29730c80d07b","Type":"ContainerStarted","Data":"e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7"}
Jul 02 14:38:46.010099 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.010067    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:46.010099 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.010089    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" event={"ID":"b869f464-c7f0-4c3a-9a25-29730c80d07b","Type":"ContainerStarted","Data":"2e8074de72b1ce5e8de635a1f76399aa77c25182e5c810251e110f4e466e4213"}
Jul 02 14:38:46.030734 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.030695    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podStartSLOduration=2.030682219 podStartE2EDuration="2.030682219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:38:46.029075473 +0000 UTC m=+1332.792460791" watchObservedRunningTime="2026-07-02 14:38:46.030682219 +0000 UTC m=+1332.794067538"
Jul 02 14:38:46.054813 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:46.054768    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podStartSLOduration=2.054753905 podStartE2EDuration="2.054753905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:38:46.053509001 +0000 UTC m=+1332.816894316" watchObservedRunningTime="2026-07-02 14:38:46.054753905 +0000 UTC m=+1332.818139226"
Jul 02 14:38:47.014349 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:47.014304    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:38:47.014349 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:47.014329    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:47.015513 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:47.015487    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:38:48.018064 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.018026    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:38:48.645417 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.645367    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 02 14:38:48.649530 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.649487    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 02 14:38:48.748128 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.748103    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:38:48.751112 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.751094    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:38:48.848151 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.848112    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-api-access-qcw82\") pod \"130d2b67-9d70-4a56-b596-122ef94e67cf\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") "
Jul 02 14:38:48.848298 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.848172    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-rbac-proxy-sar-config\") pod \"adc7e05a-8111-434b-8c37-8804fa0c10f4\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") "
Jul 02 14:38:48.848298 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.848200    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-rbac-proxy-sar-config\") pod \"130d2b67-9d70-4a56-b596-122ef94e67cf\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") "
Jul 02 14:38:48.848298 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.848225    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130d2b67-9d70-4a56-b596-122ef94e67cf-proxy-tls\") pod \"130d2b67-9d70-4a56-b596-122ef94e67cf\" (UID: \"130d2b67-9d70-4a56-b596-122ef94e67cf\") "
Jul 02 14:38:48.848298 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.848287    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adc7e05a-8111-434b-8c37-8804fa0c10f4-proxy-tls\") pod \"adc7e05a-8111-434b-8c37-8804fa0c10f4\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") "
Jul 02 14:38:48.848464 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.848310    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfrpc\" (UniqueName: \"kubernetes.io/projected/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-api-access-jfrpc\") pod \"adc7e05a-8111-434b-8c37-8804fa0c10f4\" (UID: \"adc7e05a-8111-434b-8c37-8804fa0c10f4\") "
Jul 02 14:38:48.848658 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.848628    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "130d2b67-9d70-4a56-b596-122ef94e67cf" (UID: "130d2b67-9d70-4a56-b596-122ef94e67cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:38:48.848740 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.848643    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "adc7e05a-8111-434b-8c37-8804fa0c10f4" (UID: "adc7e05a-8111-434b-8c37-8804fa0c10f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:38:48.850414 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.850384    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc7e05a-8111-434b-8c37-8804fa0c10f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "adc7e05a-8111-434b-8c37-8804fa0c10f4" (UID: "adc7e05a-8111-434b-8c37-8804fa0c10f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:38:48.850414 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.850395    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-api-access-qcw82" (OuterVolumeSpecName: "kube-api-access-qcw82") pod "130d2b67-9d70-4a56-b596-122ef94e67cf" (UID: "130d2b67-9d70-4a56-b596-122ef94e67cf"). InnerVolumeSpecName "kube-api-access-qcw82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:38:48.850414 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.850389    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130d2b67-9d70-4a56-b596-122ef94e67cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "130d2b67-9d70-4a56-b596-122ef94e67cf" (UID: "130d2b67-9d70-4a56-b596-122ef94e67cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:38:48.850642 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.850440    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-api-access-jfrpc" (OuterVolumeSpecName: "kube-api-access-jfrpc") pod "adc7e05a-8111-434b-8c37-8804fa0c10f4" (UID: "adc7e05a-8111-434b-8c37-8804fa0c10f4"). InnerVolumeSpecName "kube-api-access-jfrpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:38:48.949440 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.949399    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:38:48.949440 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.949432    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:38:48.949440 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.949443    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/130d2b67-9d70-4a56-b596-122ef94e67cf-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:38:48.949440 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.949452    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adc7e05a-8111-434b-8c37-8804fa0c10f4-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:38:48.949744 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.949460    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfrpc\" (UniqueName: \"kubernetes.io/projected/adc7e05a-8111-434b-8c37-8804fa0c10f4-kube-api-access-jfrpc\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:38:48.949744 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:48.949493    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcw82\" (UniqueName: \"kubernetes.io/projected/130d2b67-9d70-4a56-b596-122ef94e67cf-kube-api-access-qcw82\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:38:49.023670 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.023636    2551 generic.go:356] "Generic (PLEG): container finished" podID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerID="fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f" exitCode=0
Jul 02 14:38:49.024068 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.023687    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"
Jul 02 14:38:49.024068 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.023729    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" event={"ID":"adc7e05a-8111-434b-8c37-8804fa0c10f4","Type":"ContainerDied","Data":"fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f"}
Jul 02 14:38:49.024068 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.023790    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v" event={"ID":"adc7e05a-8111-434b-8c37-8804fa0c10f4","Type":"ContainerDied","Data":"844dfca2e5f959d6a7c7518c61925d22b084401ed733b8fb26902bfa08d47e0e"}
Jul 02 14:38:49.024068 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.023833    2551 scope.go:117] "RemoveContainer" containerID="7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44"
Jul 02 14:38:49.025373 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.025352    2551 generic.go:356] "Generic (PLEG): container finished" podID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerID="9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2" exitCode=0
Jul 02 14:38:49.025449 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.025405    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"
Jul 02 14:38:49.025449 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.025415    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" event={"ID":"130d2b67-9d70-4a56-b596-122ef94e67cf","Type":"ContainerDied","Data":"9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2"}
Jul 02 14:38:49.025598 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.025494    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" event={"ID":"130d2b67-9d70-4a56-b596-122ef94e67cf","Type":"ContainerDied","Data":"e7e7cc18bc85e2f1fcd136709da6c0d7c0964ee5f1d44d66326255ed4c98272d"}
Jul 02 14:38:49.032256 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.032239    2551 scope.go:117] "RemoveContainer" containerID="fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f"
Jul 02 14:38:49.040257 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.040239    2551 scope.go:117] "RemoveContainer" containerID="7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44"
Jul 02 14:38:49.040521 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:38:49.040504    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44\": container with ID starting with 7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44 not found: ID does not exist" containerID="7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44"
Jul 02 14:38:49.040576 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.040533    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44"} err="failed to get container status \"7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44\": rpc error: code = NotFound desc = could not find container \"7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44\": container with ID starting with 7e4e831e1b80f115745750e479bcf0074a8eef71994428fe145a940df59bdb44 not found: ID does not exist"
Jul 02 14:38:49.040576 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.040550    2551 scope.go:117] "RemoveContainer" containerID="fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f"
Jul 02 14:38:49.040806 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:38:49.040788    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f\": container with ID starting with fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f not found: ID does not exist" containerID="fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f"
Jul 02 14:38:49.040870 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.040816    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f"} err="failed to get container status \"fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f\": rpc error: code = NotFound desc = could not find container \"fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f\": container with ID starting with fe936190759ed2e375d280c170e07f8b7bc39bb43909c745b63a923683a5f53f not found: ID does not exist"
Jul 02 14:38:49.040870 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.040841    2551 scope.go:117] "RemoveContainer" containerID="3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce"
Jul 02 14:38:49.048479 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.048451    2551 scope.go:117] "RemoveContainer" containerID="9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2"
Jul 02 14:38:49.052283 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.052264    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"]
Jul 02 14:38:49.056246 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.056231    2551 scope.go:117] "RemoveContainer" containerID="3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce"
Jul 02 14:38:49.056519 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:38:49.056498    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce\": container with ID starting with 3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce not found: ID does not exist" containerID="3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce"
Jul 02 14:38:49.056576 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.056525    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce"} err="failed to get container status \"3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce\": rpc error: code = NotFound desc = could not find container \"3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce\": container with ID starting with 3f3eb26633238b4f0d445277934a4c8519774d7c8ed27524b0d164ec2cb45bce not found: ID does not exist"
Jul 02 14:38:49.056576 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.056542    2551 scope.go:117] "RemoveContainer" containerID="9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2"
Jul 02 14:38:49.056727 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.056710    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h"]
Jul 02 14:38:49.056787 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:38:49.056769    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2\": container with ID starting with 9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2 not found: ID does not exist" containerID="9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2"
Jul 02 14:38:49.056826 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.056786    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2"} err="failed to get container status \"9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2\": rpc error: code = NotFound desc = could not find container \"9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2\": container with ID starting with 9d72e14f7bec319b12ce19f458d579a86f647185211774bbde89e4b6d8923ed2 not found: ID does not exist"
Jul 02 14:38:49.070766 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.070744    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"]
Jul 02 14:38:49.078465 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.078446    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8937c-predictor-f4ccb5bcf-n2l7v"]
Jul 02 14:38:49.644869 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.644826    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": context deadline exceeded"
Jul 02 14:38:49.650104 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.650052    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8937c-predictor-775f4ccc46-j956h" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: i/o timeout"
Jul 02 14:38:49.753485 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.753435    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" path="/var/lib/kubelet/pods/130d2b67-9d70-4a56-b596-122ef94e67cf/volumes"
Jul 02 14:38:49.753892 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:49.753872    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" path="/var/lib/kubelet/pods/adc7e05a-8111-434b-8c37-8804fa0c10f4/volumes"
Jul 02 14:38:52.018892 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:52.018863    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:38:52.019358 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:52.019309    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:38:53.023635 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:53.023605    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:38:53.024152 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:38:53.024124    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:39:02.019445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:39:02.019367    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:39:03.024992 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:39:03.024947    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:39:12.020299 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:39:12.020260    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:39:13.024450 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:39:13.024407    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:39:22.019445 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:39:22.019402    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:39:23.024617 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:39:23.024565    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:39:32.020665 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:39:32.020633    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:39:33.024666 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:39:33.024634    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:41:33.803095 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:41:33.803069    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:41:33.804528 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:41:33.804506    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:46:33.825347 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:33.825318    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:46:33.827771 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:33.827755    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:46:49.608672 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.608642    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"]
Jul 02 14:46:49.609087 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.608864    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container" containerID="cri-o://b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567" gracePeriod=30
Jul 02 14:46:49.609087 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.608905    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kube-rbac-proxy" containerID="cri-o://c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e" gracePeriod=30
Jul 02 14:46:49.666362 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666327    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"]
Jul 02 14:46:49.666752 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666736    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kube-rbac-proxy"
Jul 02 14:46:49.666822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666754    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kube-rbac-proxy"
Jul 02 14:46:49.666822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666769    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container"
Jul 02 14:46:49.666822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666775    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container"
Jul 02 14:46:49.666822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666793    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kube-rbac-proxy"
Jul 02 14:46:49.666822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666799    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kube-rbac-proxy"
Jul 02 14:46:49.666822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666806    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container"
Jul 02 14:46:49.666822 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666812    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container"
Jul 02 14:46:49.667160 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666860    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kserve-container"
Jul 02 14:46:49.667160 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666867    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kserve-container"
Jul 02 14:46:49.667160 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666874    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="130d2b67-9d70-4a56-b596-122ef94e67cf" containerName="kube-rbac-proxy"
Jul 02 14:46:49.667160 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.666881    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="adc7e05a-8111-434b-8c37-8804fa0c10f4" containerName="kube-rbac-proxy"
Jul 02 14:46:49.669754 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.669738    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:49.672458 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.672440    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-484ca-predictor-serving-cert\""
Jul 02 14:46:49.672582 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.672462    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-484ca-kube-rbac-proxy-sar-config\""
Jul 02 14:46:49.680528 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.680504    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"]
Jul 02 14:46:49.739000 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.738968    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"]
Jul 02 14:46:49.742146 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.742123    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:49.744893 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.744867    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-484ca-kube-rbac-proxy-sar-config\""
Jul 02 14:46:49.745007 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.744935    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-484ca-predictor-serving-cert\""
Jul 02 14:46:49.754401 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.754377    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"]
Jul 02 14:46:49.783285 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.783255    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"]
Jul 02 14:46:49.783805 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.783559    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container" containerID="cri-o://261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812" gracePeriod=30
Jul 02 14:46:49.783805 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.783697    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kube-rbac-proxy" containerID="cri-o://35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d" gracePeriod=30
Jul 02 14:46:49.790978 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.790938    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:49.791245 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.791188    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fk8\" (UniqueName: \"kubernetes.io/projected/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-api-access-t7fk8\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:49.791465 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.791403    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-proxy-tls\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:49.854407 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.854375    2551 generic.go:356] "Generic (PLEG): container finished" podID="9b829695-5466-4067-a183-a1a7434cb2c8" containerID="c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e" exitCode=2
Jul 02 14:46:49.854547 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.854445    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" event={"ID":"9b829695-5466-4067-a183-a1a7434cb2c8","Type":"ContainerDied","Data":"c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e"}
Jul 02 14:46:49.892158 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.892079    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:49.892158 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.892128    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67wh\" (UniqueName: \"kubernetes.io/projected/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-api-access-j67wh\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:49.892343 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.892168    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-proxy-tls\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:49.892343 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.892235    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fk8\" (UniqueName: \"kubernetes.io/projected/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-api-access-t7fk8\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:49.892343 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.892265    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:49.892343 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.892295    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-proxy-tls\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:49.892602 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:46:49.892513    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-484ca-predictor-serving-cert: secret "success-200-isvc-484ca-predictor-serving-cert" not found
Jul 02 14:46:49.892602 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:46:49.892578    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-proxy-tls podName:03b37e0d-c12b-4e34-bcb5-bf622dee8b92 nodeName:}" failed. No retries permitted until 2026-07-02 14:46:50.39255812 +0000 UTC m=+1817.155943420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-proxy-tls") pod "success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" (UID: "03b37e0d-c12b-4e34-bcb5-bf622dee8b92") : secret "success-200-isvc-484ca-predictor-serving-cert" not found
Jul 02 14:46:49.893407 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.893386    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:49.901602 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.901581    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fk8\" (UniqueName: \"kubernetes.io/projected/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-api-access-t7fk8\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:49.993706 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.993668    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67wh\" (UniqueName: \"kubernetes.io/projected/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-api-access-j67wh\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:49.993706 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.993703    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-proxy-tls\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:49.993916 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.993742    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:49.993916 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:46:49.993860    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-484ca-predictor-serving-cert: secret "error-404-isvc-484ca-predictor-serving-cert" not found
Jul 02 14:46:49.994063 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:46:49.993925    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-proxy-tls podName:ce9256f2-780e-40cc-bfb8-343cbdf42f3f nodeName:}" failed. No retries permitted until 2026-07-02 14:46:50.49390565 +0000 UTC m=+1817.257290971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-proxy-tls") pod "error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" (UID: "ce9256f2-780e-40cc-bfb8-343cbdf42f3f") : secret "error-404-isvc-484ca-predictor-serving-cert" not found
Jul 02 14:46:49.994359 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:49.994339    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:50.006548 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.006524    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67wh\" (UniqueName: \"kubernetes.io/projected/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-api-access-j67wh\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:50.397457 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.397421    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-proxy-tls\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:50.399806 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.399781    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-proxy-tls\") pod \"success-200-isvc-484ca-predictor-6c98768cdb-w6x8j\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") " pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:50.498712 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.498689    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-proxy-tls\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:50.500899 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.500873    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-proxy-tls\") pod \"error-404-isvc-484ca-predictor-9b4fd8447-xvlc9\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") " pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:50.580968 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.580943    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:50.655827 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.655759    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:50.722345 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.722319    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"]
Jul 02 14:46:50.723965 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:46:50.723936    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b37e0d_c12b_4e34_bcb5_bf622dee8b92.slice/crio-9c0e7d7ea431586d95f7d9c69c8aa64fab292f4f83eb666677f54badc4ee6b22 WatchSource:0}: Error finding container 9c0e7d7ea431586d95f7d9c69c8aa64fab292f4f83eb666677f54badc4ee6b22: Status 404 returned error can't find the container with id 9c0e7d7ea431586d95f7d9c69c8aa64fab292f4f83eb666677f54badc4ee6b22
Jul 02 14:46:50.800887 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.800859    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"]
Jul 02 14:46:50.803107 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:46:50.803080    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9256f2_780e_40cc_bfb8_343cbdf42f3f.slice/crio-e2f6f9d9093ab1e619cd1329b79040de2a7b52d7f4c662dbf92cc58cde426da8 WatchSource:0}: Error finding container e2f6f9d9093ab1e619cd1329b79040de2a7b52d7f4c662dbf92cc58cde426da8: Status 404 returned error can't find the container with id e2f6f9d9093ab1e619cd1329b79040de2a7b52d7f4c662dbf92cc58cde426da8
Jul 02 14:46:50.861055 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.861030    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" event={"ID":"ce9256f2-780e-40cc-bfb8-343cbdf42f3f","Type":"ContainerStarted","Data":"e2f6f9d9093ab1e619cd1329b79040de2a7b52d7f4c662dbf92cc58cde426da8"}
Jul 02 14:46:50.862984 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.862960    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" event={"ID":"03b37e0d-c12b-4e34-bcb5-bf622dee8b92","Type":"ContainerStarted","Data":"33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047"}
Jul 02 14:46:50.863071 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.863018    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" event={"ID":"03b37e0d-c12b-4e34-bcb5-bf622dee8b92","Type":"ContainerStarted","Data":"9c0e7d7ea431586d95f7d9c69c8aa64fab292f4f83eb666677f54badc4ee6b22"}
Jul 02 14:46:50.864648 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.864629    2551 generic.go:356] "Generic (PLEG): container finished" podID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerID="35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d" exitCode=2
Jul 02 14:46:50.864741 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:50.864653    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" event={"ID":"8c2fe9bf-8edd-416f-b699-67c932039ec6","Type":"ContainerDied","Data":"35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d"}
Jul 02 14:46:51.870929 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.870885    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" event={"ID":"03b37e0d-c12b-4e34-bcb5-bf622dee8b92","Type":"ContainerStarted","Data":"6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb"}
Jul 02 14:46:51.871367 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.871079    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:51.871367 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.871145    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:51.872668 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.872642    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 14:46:51.872791 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.872745    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" event={"ID":"ce9256f2-780e-40cc-bfb8-343cbdf42f3f","Type":"ContainerStarted","Data":"cd2c191186b09eca78fc717a69cdde2a7c66a4d22b26395addc7f553ded4884b"}
Jul 02 14:46:51.872791 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.872786    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" event={"ID":"ce9256f2-780e-40cc-bfb8-343cbdf42f3f","Type":"ContainerStarted","Data":"e949080b1167899b7ca6446a2d76a5dece7baa6657448dfe748a7a433f821399"}
Jul 02 14:46:51.872969 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.872954    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:51.873024 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.873005    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:51.873869 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.873850    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 14:46:51.891380 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.891333    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podStartSLOduration=2.891318387 podStartE2EDuration="2.891318387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:46:51.889932914 +0000 UTC m=+1818.653318235" watchObservedRunningTime="2026-07-02 14:46:51.891318387 +0000 UTC m=+1818.654703795"
Jul 02 14:46:51.908743 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:51.908697    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podStartSLOduration=2.908683508 podStartE2EDuration="2.908683508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:46:51.907970113 +0000 UTC m=+1818.671355434" watchObservedRunningTime="2026-07-02 14:46:51.908683508 +0000 UTC m=+1818.672068829"
Jul 02 14:46:52.719281 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:52.719236    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 02 14:46:52.719439 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:52.719236    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 02 14:46:52.723611 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:52.723589    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 02 14:46:52.723661 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:52.723594    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 02 14:46:52.877176 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:52.877125    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 14:46:52.877671 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:52.877197    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 14:46:53.042828 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.042799    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:46:53.223912 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.223888    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s567n\" (UniqueName: \"kubernetes.io/projected/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-api-access-s567n\") pod \"8c2fe9bf-8edd-416f-b699-67c932039ec6\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") "
Jul 02 14:46:53.224048 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.223985    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c2fe9bf-8edd-416f-b699-67c932039ec6-proxy-tls\") pod \"8c2fe9bf-8edd-416f-b699-67c932039ec6\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") "
Jul 02 14:46:53.224048 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.224040    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-rbac-proxy-sar-config\") pod \"8c2fe9bf-8edd-416f-b699-67c932039ec6\" (UID: \"8c2fe9bf-8edd-416f-b699-67c932039ec6\") "
Jul 02 14:46:53.224430 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.224406    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c2fe9bf-8edd-416f-b699-67c932039ec6" (UID: "8c2fe9bf-8edd-416f-b699-67c932039ec6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:46:53.226876 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.226851    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-api-access-s567n" (OuterVolumeSpecName: "kube-api-access-s567n") pod "8c2fe9bf-8edd-416f-b699-67c932039ec6" (UID: "8c2fe9bf-8edd-416f-b699-67c932039ec6"). InnerVolumeSpecName "kube-api-access-s567n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:46:53.226976 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.226871    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2fe9bf-8edd-416f-b699-67c932039ec6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c2fe9bf-8edd-416f-b699-67c932039ec6" (UID: "8c2fe9bf-8edd-416f-b699-67c932039ec6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:46:53.254932 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.254913    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:46:53.325256 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.325195    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s567n\" (UniqueName: \"kubernetes.io/projected/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-api-access-s567n\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:46:53.325256 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.325218    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c2fe9bf-8edd-416f-b699-67c932039ec6-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:46:53.325256 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.325228    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c2fe9bf-8edd-416f-b699-67c932039ec6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:46:53.425813 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.425782    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b829695-5466-4067-a183-a1a7434cb2c8-proxy-tls\") pod \"9b829695-5466-4067-a183-a1a7434cb2c8\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") "
Jul 02 14:46:53.425954 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.425839    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b829695-5466-4067-a183-a1a7434cb2c8-kube-rbac-proxy-sar-config\") pod \"9b829695-5466-4067-a183-a1a7434cb2c8\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") "
Jul 02 14:46:53.425954 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.425866    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw4f2\" (UniqueName: \"kubernetes.io/projected/9b829695-5466-4067-a183-a1a7434cb2c8-kube-api-access-lw4f2\") pod \"9b829695-5466-4067-a183-a1a7434cb2c8\" (UID: \"9b829695-5466-4067-a183-a1a7434cb2c8\") "
Jul 02 14:46:53.426187 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.426165    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b829695-5466-4067-a183-a1a7434cb2c8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b829695-5466-4067-a183-a1a7434cb2c8" (UID: "9b829695-5466-4067-a183-a1a7434cb2c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:46:53.427841 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.427820    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b829695-5466-4067-a183-a1a7434cb2c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b829695-5466-4067-a183-a1a7434cb2c8" (UID: "9b829695-5466-4067-a183-a1a7434cb2c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:46:53.427935 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.427920    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b829695-5466-4067-a183-a1a7434cb2c8-kube-api-access-lw4f2" (OuterVolumeSpecName: "kube-api-access-lw4f2") pod "9b829695-5466-4067-a183-a1a7434cb2c8" (UID: "9b829695-5466-4067-a183-a1a7434cb2c8"). InnerVolumeSpecName "kube-api-access-lw4f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:46:53.527075 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.527047    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b829695-5466-4067-a183-a1a7434cb2c8-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:46:53.527075 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.527071    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b829695-5466-4067-a183-a1a7434cb2c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:46:53.527075 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.527081    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw4f2\" (UniqueName: \"kubernetes.io/projected/9b829695-5466-4067-a183-a1a7434cb2c8-kube-api-access-lw4f2\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:46:53.882728 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.882696    2551 generic.go:356] "Generic (PLEG): container finished" podID="9b829695-5466-4067-a183-a1a7434cb2c8" containerID="b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567" exitCode=0
Jul 02 14:46:53.883133 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.882746    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"
Jul 02 14:46:53.883133 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.882779    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" event={"ID":"9b829695-5466-4067-a183-a1a7434cb2c8","Type":"ContainerDied","Data":"b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567"}
Jul 02 14:46:53.883133 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.882854    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc" event={"ID":"9b829695-5466-4067-a183-a1a7434cb2c8","Type":"ContainerDied","Data":"585795edae166a343243fc10c4187a466819c8a5ef5f285ba7c35f4fb3e5cf91"}
Jul 02 14:46:53.883133 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.882893    2551 scope.go:117] "RemoveContainer" containerID="c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e"
Jul 02 14:46:53.884888 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.884863    2551 generic.go:356] "Generic (PLEG): container finished" podID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerID="261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812" exitCode=0
Jul 02 14:46:53.885002 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.884906    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"
Jul 02 14:46:53.885002 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.884913    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" event={"ID":"8c2fe9bf-8edd-416f-b699-67c932039ec6","Type":"ContainerDied","Data":"261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812"}
Jul 02 14:46:53.885002 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.884993    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx" event={"ID":"8c2fe9bf-8edd-416f-b699-67c932039ec6","Type":"ContainerDied","Data":"3422bfa1f1d02bc155cebd0dcf2c37228323f66ea0b1465346782a5a1f9d022b"}
Jul 02 14:46:53.891286 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.891267    2551 scope.go:117] "RemoveContainer" containerID="b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567"
Jul 02 14:46:53.901953 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.901928    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"]
Jul 02 14:46:53.905182 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.905160    2551 scope.go:117] "RemoveContainer" containerID="c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e"
Jul 02 14:46:53.905435 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:46:53.905416    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e\": container with ID starting with c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e not found: ID does not exist" containerID="c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e"
Jul 02 14:46:53.905558 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.905447    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e"} err="failed to get container status \"c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e\": rpc error: code = NotFound desc = could not find container \"c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e\": container with ID starting with c05e9791890954e3f1d6e129e4cfcb7f7711ff82464d47cfc65ca9a5f54ba61e not found: ID does not exist"
Jul 02 14:46:53.905558 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.905492    2551 scope.go:117] "RemoveContainer" containerID="b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567"
Jul 02 14:46:53.905757 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:46:53.905738    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567\": container with ID starting with b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567 not found: ID does not exist" containerID="b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567"
Jul 02 14:46:53.905801 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.905763    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567"} err="failed to get container status \"b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567\": rpc error: code = NotFound desc = could not find container \"b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567\": container with ID starting with b5102c009c915766f0c4012b110c9f191855d858af015c9804e57006418f8567 not found: ID does not exist"
Jul 02 14:46:53.905801 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.905777    2551 scope.go:117] "RemoveContainer" containerID="35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d"
Jul 02 14:46:53.908501 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.908465    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-57d01-predictor-7b7d9b5777-88mnx"]
Jul 02 14:46:53.913343 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.913197    2551 scope.go:117] "RemoveContainer" containerID="261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812"
Jul 02 14:46:53.920091 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.920072    2551 scope.go:117] "RemoveContainer" containerID="35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d"
Jul 02 14:46:53.920346 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:46:53.920327    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d\": container with ID starting with 35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d not found: ID does not exist" containerID="35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d"
Jul 02 14:46:53.920403 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.920353    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d"} err="failed to get container status \"35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d\": rpc error: code = NotFound desc = could not find container \"35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d\": container with ID starting with 35bb9c4dc6480c7c1a39983efa02e28d64f2bb599d0929cf7be5809af23fb80d not found: ID does not exist"
Jul 02 14:46:53.920403 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.920368    2551 scope.go:117] "RemoveContainer" containerID="261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812"
Jul 02 14:46:53.920641 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:46:53.920619    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812\": container with ID starting with 261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812 not found: ID does not exist" containerID="261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812"
Jul 02 14:46:53.920687 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.920649    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812"} err="failed to get container status \"261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812\": rpc error: code = NotFound desc = could not find container \"261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812\": container with ID starting with 261a9b11272fd6bb4b68d391f90172be8b70a3a231e164944dd2ccf2f74a6812 not found: ID does not exist"
Jul 02 14:46:53.921297 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.921279    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"]
Jul 02 14:46:53.925747 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:53.925728    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57d01-predictor-5bb7475544-nw2mc"]
Jul 02 14:46:55.748893 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:55.748856    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" path="/var/lib/kubelet/pods/8c2fe9bf-8edd-416f-b699-67c932039ec6/volumes"
Jul 02 14:46:55.749298 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:55.749284    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" path="/var/lib/kubelet/pods/9b829695-5466-4067-a183-a1a7434cb2c8/volumes"
Jul 02 14:46:57.881251 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:57.881228    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:46:57.881705 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:57.881678    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:46:57.881760 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:57.881704    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 14:46:57.882082 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:46:57.882059    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 14:47:07.881735 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:07.881691    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 14:47:07.882253 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:07.882077    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 14:47:17.881794 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:17.881749    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 14:47:17.882230 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:17.882047    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 14:47:27.881699 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:27.881659    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 02 14:47:27.882099 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:27.882071    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 02 14:47:37.882435 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:37.882402    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:47:37.882868 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:37.882718    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:47:59.336982 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.336890    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"]
Jul 02 14:47:59.337564 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.337241    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container" containerID="cri-o://e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7" gracePeriod=30
Jul 02 14:47:59.337564 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.337283    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kube-rbac-proxy" containerID="cri-o://7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408" gracePeriod=30
Jul 02 14:47:59.396145 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.396109    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"]
Jul 02 14:47:59.396425 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.396395    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container" containerID="cri-o://300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7" gracePeriod=30
Jul 02 14:47:59.396567 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.396415    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kube-rbac-proxy" containerID="cri-o://9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74" gracePeriod=30
Jul 02 14:47:59.400730 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.400702    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"]
Jul 02 14:47:59.401076 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401061    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kube-rbac-proxy"
Jul 02 14:47:59.401076 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401076    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kube-rbac-proxy"
Jul 02 14:47:59.401227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401097    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container"
Jul 02 14:47:59.401227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401106    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container"
Jul 02 14:47:59.401227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401115    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kube-rbac-proxy"
Jul 02 14:47:59.401227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401122    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kube-rbac-proxy"
Jul 02 14:47:59.401227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401134    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container"
Jul 02 14:47:59.401227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401139    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container"
Jul 02 14:47:59.401227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401209    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kube-rbac-proxy"
Jul 02 14:47:59.401227 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401222    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kube-rbac-proxy"
Jul 02 14:47:59.401517 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401235    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b829695-5466-4067-a183-a1a7434cb2c8" containerName="kserve-container"
Jul 02 14:47:59.401517 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.401246    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c2fe9bf-8edd-416f-b699-67c932039ec6" containerName="kserve-container"
Jul 02 14:47:59.404689 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.404671    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:47:59.407189 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.407170    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cb06c-kube-rbac-proxy-sar-config\""
Jul 02 14:47:59.407275 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.407259    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cb06c-predictor-serving-cert\""
Jul 02 14:47:59.414482 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.414432    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"]
Jul 02 14:47:59.509711 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.509680    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"]
Jul 02 14:47:59.513630 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.513609    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.515993 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.515970    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cb06c-predictor-serving-cert\""
Jul 02 14:47:59.516182 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.516030    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cb06c-kube-rbac-proxy-sar-config\""
Jul 02 14:47:59.524440 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.524413    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"]
Jul 02 14:47:59.542805 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.542772    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e99f649-b46f-4221-a022-a860fc8d998a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:47:59.542983 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.542900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e99f649-b46f-4221-a022-a860fc8d998a-proxy-tls\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:47:59.543053 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.542992    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw9v\" (UniqueName: \"kubernetes.io/projected/5e99f649-b46f-4221-a022-a860fc8d998a-kube-api-access-cfw9v\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:47:59.643644 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.643551    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e99f649-b46f-4221-a022-a860fc8d998a-proxy-tls\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:47:59.643644 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.643592    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") " pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.643644 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.643610    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f7nz\" (UniqueName: \"kubernetes.io/projected/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-api-access-8f7nz\") pod \"error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") " pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.643929 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.643660    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-proxy-tls\") pod \"error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") " pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.643929 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.643690    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw9v\" (UniqueName: \"kubernetes.io/projected/5e99f649-b46f-4221-a022-a860fc8d998a-kube-api-access-cfw9v\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:47:59.643929 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.643711    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e99f649-b46f-4221-a022-a860fc8d998a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:47:59.643929 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:47:59.643748    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-serving-cert: secret "success-200-isvc-cb06c-predictor-serving-cert" not found
Jul 02 14:47:59.643929 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:47:59.643822    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e99f649-b46f-4221-a022-a860fc8d998a-proxy-tls podName:5e99f649-b46f-4221-a022-a860fc8d998a nodeName:}" failed. No retries permitted until 2026-07-02 14:48:00.143799008 +0000 UTC m=+1886.907184309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5e99f649-b46f-4221-a022-a860fc8d998a-proxy-tls") pod "success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" (UID: "5e99f649-b46f-4221-a022-a860fc8d998a") : secret "success-200-isvc-cb06c-predictor-serving-cert" not found
Jul 02 14:47:59.644395 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.644375    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e99f649-b46f-4221-a022-a860fc8d998a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:47:59.654717 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.654690    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw9v\" (UniqueName: \"kubernetes.io/projected/5e99f649-b46f-4221-a022-a860fc8d998a-kube-api-access-cfw9v\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:47:59.744189 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.744153    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-proxy-tls\") pod \"error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") " pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.744422 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.744272    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") " pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.744422 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.744307    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7nz\" (UniqueName: \"kubernetes.io/projected/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-api-access-8f7nz\") pod \"error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") " pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.745057 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.745032    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") " pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.746901 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.746868    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-proxy-tls\") pod \"error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") " pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.754827 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.754800    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7nz\" (UniqueName: \"kubernetes.io/projected/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-api-access-8f7nz\") pod \"error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") " pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.825617 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.825579    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:47:59.875914 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.875319    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"]
Jul 02 14:47:59.875914 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.875611    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container" containerID="cri-o://cd2c191186b09eca78fc717a69cdde2a7c66a4d22b26395addc7f553ded4884b" gracePeriod=30
Jul 02 14:47:59.875914 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.875735    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kube-rbac-proxy" containerID="cri-o://e949080b1167899b7ca6446a2d76a5dece7baa6657448dfe748a7a433f821399" gracePeriod=30
Jul 02 14:47:59.938513 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.938101    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"]
Jul 02 14:47:59.938513 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.938190    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"]
Jul 02 14:47:59.942460 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.938924    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container" containerID="cri-o://33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047" gracePeriod=30
Jul 02 14:47:59.942460 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.939342    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kube-rbac-proxy" containerID="cri-o://6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb" gracePeriod=30
Jul 02 14:47:59.946203 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.946182    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:47:59.952697 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.952674    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2963e-predictor-serving-cert\""
Jul 02 14:47:59.953102 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.953070    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2963e-kube-rbac-proxy-sar-config\""
Jul 02 14:47:59.954793 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:47:59.954714    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"]
Jul 02 14:48:00.009418 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.009252    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"]
Jul 02 14:48:00.016008 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:48:00.015979    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467d3d03_2a76_4f21_af2d_7f0eb0ee6a3c.slice/crio-b8b0249754735102cfcbafd7b25ba7105e7f8d9d225434d3bac4c549c9ddadb1 WatchSource:0}: Error finding container b8b0249754735102cfcbafd7b25ba7105e7f8d9d225434d3bac4c549c9ddadb1: Status 404 returned error can't find the container with id b8b0249754735102cfcbafd7b25ba7105e7f8d9d225434d3bac4c549c9ddadb1
Jul 02 14:48:00.046750 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.046664    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2963e-predictor-5f9d949bd-h2vcz\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") " pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.046927 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.046824    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfl5x\" (UniqueName: \"kubernetes.io/projected/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-api-access-dfl5x\") pod \"success-200-isvc-2963e-predictor-5f9d949bd-h2vcz\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") " pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.046927 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.046876    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/389a18ea-e78b-461f-a2ec-b5071b01e38f-proxy-tls\") pod \"success-200-isvc-2963e-predictor-5f9d949bd-h2vcz\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") " pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.061733 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.061702    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"]
Jul 02 14:48:00.065497 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.065459    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.068129 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.068086    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2963e-predictor-serving-cert\""
Jul 02 14:48:00.068309 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.068175    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2963e-kube-rbac-proxy-sar-config\""
Jul 02 14:48:00.076714 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.076686    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"]
Jul 02 14:48:00.148450 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.148418    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e99f649-b46f-4221-a022-a860fc8d998a-proxy-tls\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:48:00.148614 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.148516    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2963e-predictor-5f9d949bd-h2vcz\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") " pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.148614 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.148589    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfl5x\" (UniqueName: \"kubernetes.io/projected/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-api-access-dfl5x\") pod \"success-200-isvc-2963e-predictor-5f9d949bd-h2vcz\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") " pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.148727 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.148618    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c220f358-6343-4525-880a-e0ebfb69e173-proxy-tls\") pod \"error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") " pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.148727 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.148658    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l595l\" (UniqueName: \"kubernetes.io/projected/c220f358-6343-4525-880a-e0ebfb69e173-kube-api-access-l595l\") pod \"error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") " pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.148727 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.148692    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/389a18ea-e78b-461f-a2ec-b5071b01e38f-proxy-tls\") pod \"success-200-isvc-2963e-predictor-5f9d949bd-h2vcz\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") " pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.148869 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.148758    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c220f358-6343-4525-880a-e0ebfb69e173-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") " pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.149173 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.149140    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2963e-predictor-5f9d949bd-h2vcz\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") " pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.152144 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.151676    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/389a18ea-e78b-461f-a2ec-b5071b01e38f-proxy-tls\") pod \"success-200-isvc-2963e-predictor-5f9d949bd-h2vcz\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") " pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.152144 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.152030    2551 generic.go:356] "Generic (PLEG): container finished" podID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerID="7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408" exitCode=2
Jul 02 14:48:00.152144 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.152106    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" event={"ID":"b869f464-c7f0-4c3a-9a25-29730c80d07b","Type":"ContainerDied","Data":"7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408"}
Jul 02 14:48:00.152589 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.152566    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e99f649-b46f-4221-a022-a860fc8d998a-proxy-tls\") pod \"success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") " pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:48:00.154187 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.154161    2551 generic.go:356] "Generic (PLEG): container finished" podID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerID="6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb" exitCode=2
Jul 02 14:48:00.154274 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.154234    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" event={"ID":"03b37e0d-c12b-4e34-bcb5-bf622dee8b92","Type":"ContainerDied","Data":"6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb"}
Jul 02 14:48:00.155365 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.155339    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" event={"ID":"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c","Type":"ContainerStarted","Data":"b8b0249754735102cfcbafd7b25ba7105e7f8d9d225434d3bac4c549c9ddadb1"}
Jul 02 14:48:00.156991 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.156970    2551 generic.go:356] "Generic (PLEG): container finished" podID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerID="9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74" exitCode=2
Jul 02 14:48:00.157091 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.157041    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" event={"ID":"9ee64839-23c4-465a-a0c5-e6daf981a63c","Type":"ContainerDied","Data":"9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74"}
Jul 02 14:48:00.158789 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.158762    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfl5x\" (UniqueName: \"kubernetes.io/projected/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-api-access-dfl5x\") pod \"success-200-isvc-2963e-predictor-5f9d949bd-h2vcz\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") " pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.158883 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.158808    2551 generic.go:356] "Generic (PLEG): container finished" podID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerID="e949080b1167899b7ca6446a2d76a5dece7baa6657448dfe748a7a433f821399" exitCode=2
Jul 02 14:48:00.158883 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.158850    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" event={"ID":"ce9256f2-780e-40cc-bfb8-343cbdf42f3f","Type":"ContainerDied","Data":"e949080b1167899b7ca6446a2d76a5dece7baa6657448dfe748a7a433f821399"}
Jul 02 14:48:00.249162 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.249121    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c220f358-6343-4525-880a-e0ebfb69e173-proxy-tls\") pod \"error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") " pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.249285 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.249166    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l595l\" (UniqueName: \"kubernetes.io/projected/c220f358-6343-4525-880a-e0ebfb69e173-kube-api-access-l595l\") pod \"error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") " pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.249285 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.249209    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c220f358-6343-4525-880a-e0ebfb69e173-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") " pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.249894 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.249874    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c220f358-6343-4525-880a-e0ebfb69e173-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") " pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.251551 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.251532    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c220f358-6343-4525-880a-e0ebfb69e173-proxy-tls\") pod \"error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") " pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.258141 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.258119    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l595l\" (UniqueName: \"kubernetes.io/projected/c220f358-6343-4525-880a-e0ebfb69e173-kube-api-access-l595l\") pod \"error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") " pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.262979 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.262959    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:00.322118 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.322088    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:48:00.423698 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.423661    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"]
Jul 02 14:48:00.427731 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:48:00.427696    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389a18ea_e78b_461f_a2ec_b5071b01e38f.slice/crio-aa5c387913c5795e7b2f97a254ec52b8263d7dded9ba283f7e0668029c7f17e9 WatchSource:0}: Error finding container aa5c387913c5795e7b2f97a254ec52b8263d7dded9ba283f7e0668029c7f17e9: Status 404 returned error can't find the container with id aa5c387913c5795e7b2f97a254ec52b8263d7dded9ba283f7e0668029c7f17e9
Jul 02 14:48:00.450545 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.450330    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:00.485500 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.485436    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"]
Jul 02 14:48:00.635220 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:00.635187    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"]
Jul 02 14:48:00.637939 ip-10-0-134-54 kubenswrapper[2551]: W0702 14:48:00.637905    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc220f358_6343_4525_880a_e0ebfb69e173.slice/crio-72efd6fbd4f637d1b1cbc132aa603dbc27c97a9153f57cb966aeb777247fc5c8 WatchSource:0}: Error finding container 72efd6fbd4f637d1b1cbc132aa603dbc27c97a9153f57cb966aeb777247fc5c8: Status 404 returned error can't find the container with id 72efd6fbd4f637d1b1cbc132aa603dbc27c97a9153f57cb966aeb777247fc5c8
Jul 02 14:48:01.167208 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.167170    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" event={"ID":"c220f358-6343-4525-880a-e0ebfb69e173","Type":"ContainerStarted","Data":"fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a"}
Jul 02 14:48:01.167421 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.167248    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" event={"ID":"c220f358-6343-4525-880a-e0ebfb69e173","Type":"ContainerStarted","Data":"d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14"}
Jul 02 14:48:01.167421 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.167278    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" event={"ID":"c220f358-6343-4525-880a-e0ebfb69e173","Type":"ContainerStarted","Data":"72efd6fbd4f637d1b1cbc132aa603dbc27c97a9153f57cb966aeb777247fc5c8"}
Jul 02 14:48:01.167421 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.167309    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:01.167421 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.167339    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:01.168762 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.168737    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 02 14:48:01.169121 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.169095    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" event={"ID":"389a18ea-e78b-461f-a2ec-b5071b01e38f","Type":"ContainerStarted","Data":"5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4"}
Jul 02 14:48:01.169234 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.169163    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" event={"ID":"389a18ea-e78b-461f-a2ec-b5071b01e38f","Type":"ContainerStarted","Data":"d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684"}
Jul 02 14:48:01.169234 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.169200    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" event={"ID":"389a18ea-e78b-461f-a2ec-b5071b01e38f","Type":"ContainerStarted","Data":"aa5c387913c5795e7b2f97a254ec52b8263d7dded9ba283f7e0668029c7f17e9"}
Jul 02 14:48:01.169234 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.169231    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:01.169405 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.169262    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:01.170349 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.170321    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 14:48:01.170991 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.170966    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" event={"ID":"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c","Type":"ContainerStarted","Data":"f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50"}
Jul 02 14:48:01.171090 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.171016    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:48:01.171090 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.171040    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" event={"ID":"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c","Type":"ContainerStarted","Data":"0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337"}
Jul 02 14:48:01.171090 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.171074    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:48:01.172312 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.172288    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:48:01.172846 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.172825    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" event={"ID":"5e99f649-b46f-4221-a022-a860fc8d998a","Type":"ContainerStarted","Data":"1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c"}
Jul 02 14:48:01.172940 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.172865    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" event={"ID":"5e99f649-b46f-4221-a022-a860fc8d998a","Type":"ContainerStarted","Data":"5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5"}
Jul 02 14:48:01.172940 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.172885    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" event={"ID":"5e99f649-b46f-4221-a022-a860fc8d998a","Type":"ContainerStarted","Data":"15a4654f8250208719d4ce1f38ee4e3849049761519f90e2235cf129643917b8"}
Jul 02 14:48:01.172940 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.172908    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:48:01.173078 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.172964    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:48:01.173849 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.173828    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:48:01.196584 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.196524    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podStartSLOduration=1.196502176 podStartE2EDuration="1.196502176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:48:01.187341936 +0000 UTC m=+1887.950727259" watchObservedRunningTime="2026-07-02 14:48:01.196502176 +0000 UTC m=+1887.959887497"
Jul 02 14:48:01.214207 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.214147    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podStartSLOduration=2.214128861 podStartE2EDuration="2.214128861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:48:01.21262507 +0000 UTC m=+1887.976010392" watchObservedRunningTime="2026-07-02 14:48:01.214128861 +0000 UTC m=+1887.977514182"
Jul 02 14:48:01.232370 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.232269    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podStartSLOduration=2.232250067 podStartE2EDuration="2.232250067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:48:01.231660913 +0000 UTC m=+1887.995046235" watchObservedRunningTime="2026-07-02 14:48:01.232250067 +0000 UTC m=+1887.995635388"
Jul 02 14:48:01.251306 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:01.251244    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podStartSLOduration=2.251224564 podStartE2EDuration="2.251224564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:48:01.251033078 +0000 UTC m=+1888.014418399" watchObservedRunningTime="2026-07-02 14:48:01.251224564 +0000 UTC m=+1888.014609887"
Jul 02 14:48:02.015308 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:02.015271    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 02 14:48:02.019562 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:02.019535    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 02 14:48:02.177077 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:02.177033    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:48:02.177294 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:02.177133    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 14:48:02.177294 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:02.177189    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:48:02.177294 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:02.177191    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 02 14:48:02.877924 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:02.877887    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 02 14:48:02.878119 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:02.877884    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 02 14:48:03.018512 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.018443    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 02 14:48:03.024838 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.024804    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 02 14:48:03.659374 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.659347    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:48:03.784610 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.784579    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee64839-23c4-465a-a0c5-e6daf981a63c-proxy-tls\") pod \"9ee64839-23c4-465a-a0c5-e6daf981a63c\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") "
Jul 02 14:48:03.784798 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.784625    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5kcf\" (UniqueName: \"kubernetes.io/projected/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-api-access-r5kcf\") pod \"9ee64839-23c4-465a-a0c5-e6daf981a63c\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") "
Jul 02 14:48:03.784798 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.784726    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-rbac-proxy-sar-config\") pod \"9ee64839-23c4-465a-a0c5-e6daf981a63c\" (UID: \"9ee64839-23c4-465a-a0c5-e6daf981a63c\") "
Jul 02 14:48:03.785169 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.785125    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ee64839-23c4-465a-a0c5-e6daf981a63c" (UID: "9ee64839-23c4-465a-a0c5-e6daf981a63c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:48:03.796243 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.788671    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-api-access-r5kcf" (OuterVolumeSpecName: "kube-api-access-r5kcf") pod "9ee64839-23c4-465a-a0c5-e6daf981a63c" (UID: "9ee64839-23c4-465a-a0c5-e6daf981a63c"). InnerVolumeSpecName "kube-api-access-r5kcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:48:03.796883 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.796848    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee64839-23c4-465a-a0c5-e6daf981a63c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ee64839-23c4-465a-a0c5-e6daf981a63c" (UID: "9ee64839-23c4-465a-a0c5-e6daf981a63c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:48:03.887505 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.887218    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:03.887505 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.887250    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee64839-23c4-465a-a0c5-e6daf981a63c-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:03.887505 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.887282    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5kcf\" (UniqueName: \"kubernetes.io/projected/9ee64839-23c4-465a-a0c5-e6daf981a63c-kube-api-access-r5kcf\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:03.980606 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:03.980578    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:48:04.088694 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.088652    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-448z4\" (UniqueName: \"kubernetes.io/projected/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-api-access-448z4\") pod \"b869f464-c7f0-4c3a-9a25-29730c80d07b\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") "
Jul 02 14:48:04.089197 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.088725    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-rbac-proxy-sar-config\") pod \"b869f464-c7f0-4c3a-9a25-29730c80d07b\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") "
Jul 02 14:48:04.089197 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.088760    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b869f464-c7f0-4c3a-9a25-29730c80d07b-proxy-tls\") pod \"b869f464-c7f0-4c3a-9a25-29730c80d07b\" (UID: \"b869f464-c7f0-4c3a-9a25-29730c80d07b\") "
Jul 02 14:48:04.089748 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.089710    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b869f464-c7f0-4c3a-9a25-29730c80d07b" (UID: "b869f464-c7f0-4c3a-9a25-29730c80d07b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:48:04.092300 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.092257    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b869f464-c7f0-4c3a-9a25-29730c80d07b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b869f464-c7f0-4c3a-9a25-29730c80d07b" (UID: "b869f464-c7f0-4c3a-9a25-29730c80d07b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:48:04.093383 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.093357    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-api-access-448z4" (OuterVolumeSpecName: "kube-api-access-448z4") pod "b869f464-c7f0-4c3a-9a25-29730c80d07b" (UID: "b869f464-c7f0-4c3a-9a25-29730c80d07b"). InnerVolumeSpecName "kube-api-access-448z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:48:04.189045 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.188658    2551 generic.go:356] "Generic (PLEG): container finished" podID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerID="300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7" exitCode=0
Jul 02 14:48:04.189045 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.188732    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"
Jul 02 14:48:04.189045 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.188763    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" event={"ID":"9ee64839-23c4-465a-a0c5-e6daf981a63c","Type":"ContainerDied","Data":"300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7"}
Jul 02 14:48:04.189045 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.188859    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq" event={"ID":"9ee64839-23c4-465a-a0c5-e6daf981a63c","Type":"ContainerDied","Data":"f5d55619c2adb00623d452b6fd25fb6a5a7fbfeecabda80a7a40101238090adb"}
Jul 02 14:48:04.189045 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.188919    2551 scope.go:117] "RemoveContainer" containerID="9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74"
Jul 02 14:48:04.199056 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.196968    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:04.199056 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.197007    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b869f464-c7f0-4c3a-9a25-29730c80d07b-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:04.199056 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.197038    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-448z4\" (UniqueName: \"kubernetes.io/projected/b869f464-c7f0-4c3a-9a25-29730c80d07b-kube-api-access-448z4\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:04.202063 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.202029    2551 generic.go:356] "Generic (PLEG): container finished" podID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerID="cd2c191186b09eca78fc717a69cdde2a7c66a4d22b26395addc7f553ded4884b" exitCode=0
Jul 02 14:48:04.202213 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.202161    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" event={"ID":"ce9256f2-780e-40cc-bfb8-343cbdf42f3f","Type":"ContainerDied","Data":"cd2c191186b09eca78fc717a69cdde2a7c66a4d22b26395addc7f553ded4884b"}
Jul 02 14:48:04.204454 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.204427    2551 generic.go:356] "Generic (PLEG): container finished" podID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerID="e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7" exitCode=0
Jul 02 14:48:04.204629 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.204527    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"
Jul 02 14:48:04.204629 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.204544    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" event={"ID":"b869f464-c7f0-4c3a-9a25-29730c80d07b","Type":"ContainerDied","Data":"e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7"}
Jul 02 14:48:04.204629 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.204612    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm" event={"ID":"b869f464-c7f0-4c3a-9a25-29730c80d07b","Type":"ContainerDied","Data":"2e8074de72b1ce5e8de635a1f76399aa77c25182e5c810251e110f4e466e4213"}
Jul 02 14:48:04.233280 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.233262    2551 scope.go:117] "RemoveContainer" containerID="300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7"
Jul 02 14:48:04.249538 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.249504    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"]
Jul 02 14:48:04.253942 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.253918    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3f32-predictor-8449d84b64-z62pq"]
Jul 02 14:48:04.256213 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.256194    2551 scope.go:117] "RemoveContainer" containerID="9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74"
Jul 02 14:48:04.256620 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:48:04.256589    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74\": container with ID starting with 9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74 not found: ID does not exist" containerID="9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74"
Jul 02 14:48:04.256733 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.256631    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74"} err="failed to get container status \"9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74\": rpc error: code = NotFound desc = could not find container \"9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74\": container with ID starting with 9a54dd63598ce4a17463701f9ec54f5c40fc3edf5a3f6fff8cc7f753d8ed7b74 not found: ID does not exist"
Jul 02 14:48:04.256733 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.256656    2551 scope.go:117] "RemoveContainer" containerID="300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7"
Jul 02 14:48:04.257062 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:48:04.257035    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7\": container with ID starting with 300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7 not found: ID does not exist" containerID="300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7"
Jul 02 14:48:04.257154 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.257067    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7"} err="failed to get container status \"300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7\": rpc error: code = NotFound desc = could not find container \"300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7\": container with ID starting with 300ccd57eb14ffdb7ee302d7eeadc6dbb85a7edd1c64b1f7128cf20674e82ab7 not found: ID does not exist"
Jul 02 14:48:04.257154 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.257091    2551 scope.go:117] "RemoveContainer" containerID="7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408"
Jul 02 14:48:04.266766 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.266744    2551 scope.go:117] "RemoveContainer" containerID="e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7"
Jul 02 14:48:04.267997 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.267978    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"]
Jul 02 14:48:04.273193 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.273163    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3f32-predictor-5c75d8fd6f-54vnm"]
Jul 02 14:48:04.275460 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.275442    2551 scope.go:117] "RemoveContainer" containerID="7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408"
Jul 02 14:48:04.275775 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:48:04.275754    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408\": container with ID starting with 7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408 not found: ID does not exist" containerID="7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408"
Jul 02 14:48:04.275855 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.275782    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408"} err="failed to get container status \"7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408\": rpc error: code = NotFound desc = could not find container \"7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408\": container with ID starting with 7d2aa1a2233817ab7bdd415c04e35fbb18ea561334aef98740df49cc76b87408 not found: ID does not exist"
Jul 02 14:48:04.275855 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.275802    2551 scope.go:117] "RemoveContainer" containerID="e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7"
Jul 02 14:48:04.276161 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:48:04.276135    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7\": container with ID starting with e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7 not found: ID does not exist" containerID="e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7"
Jul 02 14:48:04.276253 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.276167    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7"} err="failed to get container status \"e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7\": rpc error: code = NotFound desc = could not find container \"e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7\": container with ID starting with e1018d963b22e10540cb8c2e42195989b2f07f2c6f1e82154cd360a5a46455b7 not found: ID does not exist"
Jul 02 14:48:04.398253 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.398225    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:48:04.499320 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.499286    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-proxy-tls\") pod \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") "
Jul 02 14:48:04.499525 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.499339    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-rbac-proxy-sar-config\") pod \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") "
Jul 02 14:48:04.499525 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.499390    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7fk8\" (UniqueName: \"kubernetes.io/projected/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-api-access-t7fk8\") pod \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\" (UID: \"03b37e0d-c12b-4e34-bcb5-bf622dee8b92\") "
Jul 02 14:48:04.499769 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.499738    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03b37e0d-c12b-4e34-bcb5-bf622dee8b92" (UID: "03b37e0d-c12b-4e34-bcb5-bf622dee8b92"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:48:04.501597 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.501564    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03b37e0d-c12b-4e34-bcb5-bf622dee8b92" (UID: "03b37e0d-c12b-4e34-bcb5-bf622dee8b92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:48:04.501718 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.501597    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-api-access-t7fk8" (OuterVolumeSpecName: "kube-api-access-t7fk8") pod "03b37e0d-c12b-4e34-bcb5-bf622dee8b92" (UID: "03b37e0d-c12b-4e34-bcb5-bf622dee8b92"). InnerVolumeSpecName "kube-api-access-t7fk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:48:04.599987 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.599953    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:04.599987 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.599989    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:04.600192 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.600002    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7fk8\" (UniqueName: \"kubernetes.io/projected/03b37e0d-c12b-4e34-bcb5-bf622dee8b92-kube-api-access-t7fk8\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:04.620435 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.620411    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:48:04.801935 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.801895    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-rbac-proxy-sar-config\") pod \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") "
Jul 02 14:48:04.802124 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.801959    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-proxy-tls\") pod \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") "
Jul 02 14:48:04.802124 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.801983    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j67wh\" (UniqueName: \"kubernetes.io/projected/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-api-access-j67wh\") pod \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\" (UID: \"ce9256f2-780e-40cc-bfb8-343cbdf42f3f\") "
Jul 02 14:48:04.802338 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.802313    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce9256f2-780e-40cc-bfb8-343cbdf42f3f" (UID: "ce9256f2-780e-40cc-bfb8-343cbdf42f3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:48:04.804082 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.804057    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce9256f2-780e-40cc-bfb8-343cbdf42f3f" (UID: "ce9256f2-780e-40cc-bfb8-343cbdf42f3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:48:04.804157 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.804107    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-api-access-j67wh" (OuterVolumeSpecName: "kube-api-access-j67wh") pod "ce9256f2-780e-40cc-bfb8-343cbdf42f3f" (UID: "ce9256f2-780e-40cc-bfb8-343cbdf42f3f"). InnerVolumeSpecName "kube-api-access-j67wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:48:04.903721 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.903525    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:04.903721 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.903565    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j67wh\" (UniqueName: \"kubernetes.io/projected/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-api-access-j67wh\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:04.903721 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:04.903580    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce9256f2-780e-40cc-bfb8-343cbdf42f3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:48:05.211072 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.210981    2551 generic.go:356] "Generic (PLEG): container finished" podID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerID="33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047" exitCode=0
Jul 02 14:48:05.211072 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.211027    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"
Jul 02 14:48:05.211545 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.211065    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" event={"ID":"03b37e0d-c12b-4e34-bcb5-bf622dee8b92","Type":"ContainerDied","Data":"33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047"}
Jul 02 14:48:05.211545 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.211143    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j" event={"ID":"03b37e0d-c12b-4e34-bcb5-bf622dee8b92","Type":"ContainerDied","Data":"9c0e7d7ea431586d95f7d9c69c8aa64fab292f4f83eb666677f54badc4ee6b22"}
Jul 02 14:48:05.211545 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.211185    2551 scope.go:117] "RemoveContainer" containerID="6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb"
Jul 02 14:48:05.213797 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.213767    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"
Jul 02 14:48:05.213797 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.213771    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9" event={"ID":"ce9256f2-780e-40cc-bfb8-343cbdf42f3f","Type":"ContainerDied","Data":"e2f6f9d9093ab1e619cd1329b79040de2a7b52d7f4c662dbf92cc58cde426da8"}
Jul 02 14:48:05.224576 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.224555    2551 scope.go:117] "RemoveContainer" containerID="33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047"
Jul 02 14:48:05.232296 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.232280    2551 scope.go:117] "RemoveContainer" containerID="6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb"
Jul 02 14:48:05.232553 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:48:05.232532    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb\": container with ID starting with 6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb not found: ID does not exist" containerID="6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb"
Jul 02 14:48:05.232630 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.232566    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb"} err="failed to get container status \"6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb\": rpc error: code = NotFound desc = could not find container \"6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb\": container with ID starting with 6caf152208c02609fccd743e2c4d4da96b55292602acbc11776b0ecd44f573eb not found: ID does not exist"
Jul 02 14:48:05.232630 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.232589    2551 scope.go:117] "RemoveContainer" containerID="33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047"
Jul 02 14:48:05.232819 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:48:05.232804    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047\": container with ID starting with 33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047 not found: ID does not exist" containerID="33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047"
Jul 02 14:48:05.232866 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.232823    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047"} err="failed to get container status \"33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047\": rpc error: code = NotFound desc = could not find container \"33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047\": container with ID starting with 33095f48ae553f8c1900defceb3bf4b640ddc92d3c8616106a4e98d72be1c047 not found: ID does not exist"
Jul 02 14:48:05.232866 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.232836    2551 scope.go:117] "RemoveContainer" containerID="e949080b1167899b7ca6446a2d76a5dece7baa6657448dfe748a7a433f821399"
Jul 02 14:48:05.238620 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.238599    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"]
Jul 02 14:48:05.240759 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.240735    2551 scope.go:117] "RemoveContainer" containerID="cd2c191186b09eca78fc717a69cdde2a7c66a4d22b26395addc7f553ded4884b"
Jul 02 14:48:05.243006 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.242985    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-484ca-predictor-6c98768cdb-w6x8j"]
Jul 02 14:48:05.256234 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.256205    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"]
Jul 02 14:48:05.261580 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.261558    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-484ca-predictor-9b4fd8447-xvlc9"]
Jul 02 14:48:05.749290 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.749258    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" path="/var/lib/kubelet/pods/03b37e0d-c12b-4e34-bcb5-bf622dee8b92/volumes"
Jul 02 14:48:05.749702 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.749689    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" path="/var/lib/kubelet/pods/9ee64839-23c4-465a-a0c5-e6daf981a63c/volumes"
Jul 02 14:48:05.750048 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.750037    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" path="/var/lib/kubelet/pods/b869f464-c7f0-4c3a-9a25-29730c80d07b/volumes"
Jul 02 14:48:05.750719 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:05.750704    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" path="/var/lib/kubelet/pods/ce9256f2-780e-40cc-bfb8-343cbdf42f3f/volumes"
Jul 02 14:48:07.181344 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:07.181317    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:48:07.181837 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:07.181768    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:48:07.181910 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:07.181832    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:48:07.182244 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:07.182217    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:48:07.182435 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:07.182420    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:07.182857 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:07.182836    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:48:07.182949 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:07.182863    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 14:48:07.183170 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:07.183153    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 02 14:48:17.181870 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:17.181831    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:48:17.182259 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:17.182226    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:48:17.183441 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:17.183417    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 02 14:48:17.183531 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:17.183439    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 14:48:27.182418 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:27.182382    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:48:27.182845 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:27.182384    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:48:27.182845 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:27.182819    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 14:48:27.184031 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:27.184009    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 02 14:48:37.181744 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:37.181701    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:48:37.182179 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:37.182151    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:48:37.183355 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:37.183332    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 02 14:48:37.183409 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:37.183366    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 14:48:47.181960 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:47.181916    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 02 14:48:47.182442 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:47.182210    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 02 14:48:47.183411 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:47.183386    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 14:48:47.183458 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:47.183390    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 02 14:48:57.182313 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:57.182279    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 14:48:57.184791 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:57.183051    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 14:48:57.184791 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:57.183322    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:48:57.184791 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:48:57.183568    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:51:33.848747 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:51:33.848719    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:51:33.851439 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:51:33.851410    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:56:33.874922 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:56:33.874893    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:56:33.877811 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:56:33.877790    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 14:57:24.744191 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:24.744116    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"]
Jul 02 14:57:24.744724 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:24.744332    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container" containerID="cri-o://d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684" gracePeriod=30
Jul 02 14:57:24.744724 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:24.744356    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kube-rbac-proxy" containerID="cri-o://5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4" gracePeriod=30
Jul 02 14:57:24.780349 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:24.780319    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"]
Jul 02 14:57:24.780590 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:24.780551    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container" containerID="cri-o://d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14" gracePeriod=30
Jul 02 14:57:24.780666 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:24.780594    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kube-rbac-proxy" containerID="cri-o://fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a" gracePeriod=30
Jul 02 14:57:25.394645 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:25.394605    2551 generic.go:356] "Generic (PLEG): container finished" podID="c220f358-6343-4525-880a-e0ebfb69e173" containerID="fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a" exitCode=2
Jul 02 14:57:25.394845 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:25.394676    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" event={"ID":"c220f358-6343-4525-880a-e0ebfb69e173","Type":"ContainerDied","Data":"fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a"}
Jul 02 14:57:25.396618 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:25.396593    2551 generic.go:356] "Generic (PLEG): container finished" podID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerID="5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4" exitCode=2
Jul 02 14:57:25.396749 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:25.396654    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" event={"ID":"389a18ea-e78b-461f-a2ec-b5071b01e38f","Type":"ContainerDied","Data":"5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4"}
Jul 02 14:57:27.177636 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:27.177553    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 02 14:57:27.177636 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:27.177552    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 02 14:57:27.183510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:27.183481    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 02 14:57:27.183510 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:27.183500    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 02 14:57:28.049223 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.049205    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:57:28.051966 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.051946    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:57:28.061944 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.061917    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c220f358-6343-4525-880a-e0ebfb69e173-kube-rbac-proxy-sar-config\") pod \"c220f358-6343-4525-880a-e0ebfb69e173\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") "
Jul 02 14:57:28.062042 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.061960    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfl5x\" (UniqueName: \"kubernetes.io/projected/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-api-access-dfl5x\") pod \"389a18ea-e78b-461f-a2ec-b5071b01e38f\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") "
Jul 02 14:57:28.062042 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.061988    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/389a18ea-e78b-461f-a2ec-b5071b01e38f-proxy-tls\") pod \"389a18ea-e78b-461f-a2ec-b5071b01e38f\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") "
Jul 02 14:57:28.062042 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.062013    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-rbac-proxy-sar-config\") pod \"389a18ea-e78b-461f-a2ec-b5071b01e38f\" (UID: \"389a18ea-e78b-461f-a2ec-b5071b01e38f\") "
Jul 02 14:57:28.062211 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.062055    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c220f358-6343-4525-880a-e0ebfb69e173-proxy-tls\") pod \"c220f358-6343-4525-880a-e0ebfb69e173\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") "
Jul 02 14:57:28.062211 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.062097    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l595l\" (UniqueName: \"kubernetes.io/projected/c220f358-6343-4525-880a-e0ebfb69e173-kube-api-access-l595l\") pod \"c220f358-6343-4525-880a-e0ebfb69e173\" (UID: \"c220f358-6343-4525-880a-e0ebfb69e173\") "
Jul 02 14:57:28.062316 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.062222    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c220f358-6343-4525-880a-e0ebfb69e173-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c220f358-6343-4525-880a-e0ebfb69e173" (UID: "c220f358-6343-4525-880a-e0ebfb69e173"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:57:28.062373 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.062325    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c220f358-6343-4525-880a-e0ebfb69e173-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:57:28.062425 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.062379    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "389a18ea-e78b-461f-a2ec-b5071b01e38f" (UID: "389a18ea-e78b-461f-a2ec-b5071b01e38f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:57:28.064080 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.064058    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-api-access-dfl5x" (OuterVolumeSpecName: "kube-api-access-dfl5x") pod "389a18ea-e78b-461f-a2ec-b5071b01e38f" (UID: "389a18ea-e78b-461f-a2ec-b5071b01e38f"). InnerVolumeSpecName "kube-api-access-dfl5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:57:28.064080 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.064077    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389a18ea-e78b-461f-a2ec-b5071b01e38f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "389a18ea-e78b-461f-a2ec-b5071b01e38f" (UID: "389a18ea-e78b-461f-a2ec-b5071b01e38f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:57:28.064207 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.064060    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c220f358-6343-4525-880a-e0ebfb69e173-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c220f358-6343-4525-880a-e0ebfb69e173" (UID: "c220f358-6343-4525-880a-e0ebfb69e173"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:57:28.064207 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.064033    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c220f358-6343-4525-880a-e0ebfb69e173-kube-api-access-l595l" (OuterVolumeSpecName: "kube-api-access-l595l") pod "c220f358-6343-4525-880a-e0ebfb69e173" (UID: "c220f358-6343-4525-880a-e0ebfb69e173"). InnerVolumeSpecName "kube-api-access-l595l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:57:28.163388 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.163356    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c220f358-6343-4525-880a-e0ebfb69e173-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:57:28.163388 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.163384    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l595l\" (UniqueName: \"kubernetes.io/projected/c220f358-6343-4525-880a-e0ebfb69e173-kube-api-access-l595l\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:57:28.163388 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.163394    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfl5x\" (UniqueName: \"kubernetes.io/projected/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-api-access-dfl5x\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:57:28.163637 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.163405    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/389a18ea-e78b-461f-a2ec-b5071b01e38f-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:57:28.163637 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.163418    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389a18ea-e78b-461f-a2ec-b5071b01e38f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 14:57:28.410670 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.410580    2551 generic.go:356] "Generic (PLEG): container finished" podID="c220f358-6343-4525-880a-e0ebfb69e173" containerID="d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14" exitCode=0
Jul 02 14:57:28.410670 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.410627    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"
Jul 02 14:57:28.411169 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.410666    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" event={"ID":"c220f358-6343-4525-880a-e0ebfb69e173","Type":"ContainerDied","Data":"d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14"}
Jul 02 14:57:28.411169 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.410735    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb" event={"ID":"c220f358-6343-4525-880a-e0ebfb69e173","Type":"ContainerDied","Data":"72efd6fbd4f637d1b1cbc132aa603dbc27c97a9153f57cb966aeb777247fc5c8"}
Jul 02 14:57:28.411169 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.410776    2551 scope.go:117] "RemoveContainer" containerID="fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a"
Jul 02 14:57:28.412340 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.412317    2551 generic.go:356] "Generic (PLEG): container finished" podID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerID="d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684" exitCode=0
Jul 02 14:57:28.412437 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.412356    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" event={"ID":"389a18ea-e78b-461f-a2ec-b5071b01e38f","Type":"ContainerDied","Data":"d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684"}
Jul 02 14:57:28.412437 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.412379    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"
Jul 02 14:57:28.412437 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.412406    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz" event={"ID":"389a18ea-e78b-461f-a2ec-b5071b01e38f","Type":"ContainerDied","Data":"aa5c387913c5795e7b2f97a254ec52b8263d7dded9ba283f7e0668029c7f17e9"}
Jul 02 14:57:28.420235 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.420101    2551 scope.go:117] "RemoveContainer" containerID="d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14"
Jul 02 14:57:28.428079 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.428063    2551 scope.go:117] "RemoveContainer" containerID="fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a"
Jul 02 14:57:28.428357 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:57:28.428336    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a\": container with ID starting with fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a not found: ID does not exist" containerID="fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a"
Jul 02 14:57:28.428411 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.428363    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a"} err="failed to get container status \"fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a\": rpc error: code = NotFound desc = could not find container \"fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a\": container with ID starting with fb5fd5cae090a9e3d627fa60f87fc25896ed32e6b3142fb56e760e2bb8913a9a not found: ID does not exist"
Jul 02 14:57:28.428411 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.428378    2551 scope.go:117] "RemoveContainer" containerID="d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14"
Jul 02 14:57:28.428649 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:57:28.428631    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14\": container with ID starting with d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14 not found: ID does not exist" containerID="d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14"
Jul 02 14:57:28.428687 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.428655    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14"} err="failed to get container status \"d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14\": rpc error: code = NotFound desc = could not find container \"d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14\": container with ID starting with d7c22543fb86f34d2cfbc257a5ab436adf6332af5a5d35e81fc08899f0b29f14 not found: ID does not exist"
Jul 02 14:57:28.428687 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.428671    2551 scope.go:117] "RemoveContainer" containerID="5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4"
Jul 02 14:57:28.435669 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.435641    2551 scope.go:117] "RemoveContainer" containerID="d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684"
Jul 02 14:57:28.440189 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.440165    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"]
Jul 02 14:57:28.443091 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.443076    2551 scope.go:117] "RemoveContainer" containerID="5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4"
Jul 02 14:57:28.443346 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:57:28.443328    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4\": container with ID starting with 5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4 not found: ID does not exist" containerID="5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4"
Jul 02 14:57:28.443388 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.443359    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4"} err="failed to get container status \"5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4\": rpc error: code = NotFound desc = could not find container \"5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4\": container with ID starting with 5577fb0f517160e53f9c38d4b11b4934654f1febd8162e61bf75b28a81c32ee4 not found: ID does not exist"
Jul 02 14:57:28.443388 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.443375    2551 scope.go:117] "RemoveContainer" containerID="d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684"
Jul 02 14:57:28.443625 ip-10-0-134-54 kubenswrapper[2551]: E0702 14:57:28.443609    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684\": container with ID starting with d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684 not found: ID does not exist" containerID="d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684"
Jul 02 14:57:28.443675 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.443627    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684"} err="failed to get container status \"d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684\": rpc error: code = NotFound desc = could not find container \"d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684\": container with ID starting with d99c492c02e515aa11cf4dc513e3896bdb6d7006e9c1861ee4d04ce7d9524684 not found: ID does not exist"
Jul 02 14:57:28.447734 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.447715    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2963e-predictor-5f9d949bd-h2vcz"]
Jul 02 14:57:28.468357 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.464449    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"]
Jul 02 14:57:28.470714 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:28.470690    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2963e-predictor-6bb9dd4685-ds5fb"]
Jul 02 14:57:29.749004 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:29.748972    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" path="/var/lib/kubelet/pods/389a18ea-e78b-461f-a2ec-b5071b01e38f/volumes"
Jul 02 14:57:29.749400 ip-10-0-134-54 kubenswrapper[2551]: I0702 14:57:29.749374    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c220f358-6343-4525-880a-e0ebfb69e173" path="/var/lib/kubelet/pods/c220f358-6343-4525-880a-e0ebfb69e173/volumes"
Jul 02 15:01:33.897835 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:01:33.897807    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 15:01:33.900389 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:01:33.900370    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 15:05:28.997962 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:28.997926    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"]
Jul 02 15:05:28.998428 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:28.998148    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container" containerID="cri-o://5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5" gracePeriod=30
Jul 02 15:05:28.998428 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:28.998173    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kube-rbac-proxy" containerID="cri-o://1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c" gracePeriod=30
Jul 02 15:05:29.063157 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:29.063129    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"]
Jul 02 15:05:29.063347 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:29.063324    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container" containerID="cri-o://0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337" gracePeriod=30
Jul 02 15:05:29.063413 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:29.063356    2551 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kube-rbac-proxy" containerID="cri-o://f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50" gracePeriod=30
Jul 02 15:05:29.184854 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:29.184828    2551 generic.go:356] "Generic (PLEG): container finished" podID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerID="f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50" exitCode=2
Jul 02 15:05:29.184946 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:29.184903    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" event={"ID":"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c","Type":"ContainerDied","Data":"f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50"}
Jul 02 15:05:29.191596 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:29.191563    2551 generic.go:356] "Generic (PLEG): container finished" podID="5e99f649-b46f-4221-a022-a860fc8d998a" containerID="1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c" exitCode=2
Jul 02 15:05:29.191684 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:29.191600    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" event={"ID":"5e99f649-b46f-4221-a022-a860fc8d998a","Type":"ContainerDied","Data":"1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c"}
Jul 02 15:05:32.024707 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.024684    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 15:05:32.027252 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.027235    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 15:05:32.125294 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.125219    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfw9v\" (UniqueName: \"kubernetes.io/projected/5e99f649-b46f-4221-a022-a860fc8d998a-kube-api-access-cfw9v\") pod \"5e99f649-b46f-4221-a022-a860fc8d998a\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") "
Jul 02 15:05:32.125294 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.125271    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e99f649-b46f-4221-a022-a860fc8d998a-proxy-tls\") pod \"5e99f649-b46f-4221-a022-a860fc8d998a\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") "
Jul 02 15:05:32.125540 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.125327    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e99f649-b46f-4221-a022-a860fc8d998a-kube-rbac-proxy-sar-config\") pod \"5e99f649-b46f-4221-a022-a860fc8d998a\" (UID: \"5e99f649-b46f-4221-a022-a860fc8d998a\") "
Jul 02 15:05:32.125731 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.125706    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e99f649-b46f-4221-a022-a860fc8d998a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e99f649-b46f-4221-a022-a860fc8d998a" (UID: "5e99f649-b46f-4221-a022-a860fc8d998a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 15:05:32.127338 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.127316    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e99f649-b46f-4221-a022-a860fc8d998a-kube-api-access-cfw9v" (OuterVolumeSpecName: "kube-api-access-cfw9v") pod "5e99f649-b46f-4221-a022-a860fc8d998a" (UID: "5e99f649-b46f-4221-a022-a860fc8d998a"). InnerVolumeSpecName "kube-api-access-cfw9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 15:05:32.127418 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.127337    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e99f649-b46f-4221-a022-a860fc8d998a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e99f649-b46f-4221-a022-a860fc8d998a" (UID: "5e99f649-b46f-4221-a022-a860fc8d998a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 15:05:32.203855 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.203828    2551 generic.go:356] "Generic (PLEG): container finished" podID="5e99f649-b46f-4221-a022-a860fc8d998a" containerID="5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5" exitCode=0
Jul 02 15:05:32.203973 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.203871    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"
Jul 02 15:05:32.203973 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.203914    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" event={"ID":"5e99f649-b46f-4221-a022-a860fc8d998a","Type":"ContainerDied","Data":"5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5"}
Jul 02 15:05:32.204091 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.203984    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz" event={"ID":"5e99f649-b46f-4221-a022-a860fc8d998a","Type":"ContainerDied","Data":"15a4654f8250208719d4ce1f38ee4e3849049761519f90e2235cf129643917b8"}
Jul 02 15:05:32.204091 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.204022    2551 scope.go:117] "RemoveContainer" containerID="1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c"
Jul 02 15:05:32.205523 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.205496    2551 generic.go:356] "Generic (PLEG): container finished" podID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerID="0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337" exitCode=0
Jul 02 15:05:32.205641 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.205525    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"
Jul 02 15:05:32.205641 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.205536    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" event={"ID":"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c","Type":"ContainerDied","Data":"0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337"}
Jul 02 15:05:32.205641 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.205587    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47" event={"ID":"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c","Type":"ContainerDied","Data":"b8b0249754735102cfcbafd7b25ba7105e7f8d9d225434d3bac4c549c9ddadb1"}
Jul 02 15:05:32.212639 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.212622    2551 scope.go:117] "RemoveContainer" containerID="5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5"
Jul 02 15:05:32.219395 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.219378    2551 scope.go:117] "RemoveContainer" containerID="1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c"
Jul 02 15:05:32.219650 ip-10-0-134-54 kubenswrapper[2551]: E0702 15:05:32.219634    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c\": container with ID starting with 1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c not found: ID does not exist" containerID="1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c"
Jul 02 15:05:32.219697 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.219657    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c"} err="failed to get container status \"1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c\": rpc error: code = NotFound desc = could not find container \"1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c\": container with ID starting with 1ee0edfb70829a82238256b369b3baf8441344aa66025735bceabab94f981b7c not found: ID does not exist"
Jul 02 15:05:32.219697 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.219673    2551 scope.go:117] "RemoveContainer" containerID="5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5"
Jul 02 15:05:32.219890 ip-10-0-134-54 kubenswrapper[2551]: E0702 15:05:32.219872    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5\": container with ID starting with 5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5 not found: ID does not exist" containerID="5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5"
Jul 02 15:05:32.219959 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.219898    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5"} err="failed to get container status \"5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5\": rpc error: code = NotFound desc = could not find container \"5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5\": container with ID starting with 5b7b5a901841d56ed3ebd4451b5242094b82f176fa273f92c3615e3509e81ba5 not found: ID does not exist"
Jul 02 15:05:32.219959 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.219919    2551 scope.go:117] "RemoveContainer" containerID="f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50"
Jul 02 15:05:32.225987 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.225962    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f7nz\" (UniqueName: \"kubernetes.io/projected/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-api-access-8f7nz\") pod \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") "
Jul 02 15:05:32.226148 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.226038    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-rbac-proxy-sar-config\") pod \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") "
Jul 02 15:05:32.226148 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.226081    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-proxy-tls\") pod \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\" (UID: \"467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c\") "
Jul 02 15:05:32.226341 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.226283    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e99f649-b46f-4221-a022-a860fc8d998a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 15:05:32.226341 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.226301    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfw9v\" (UniqueName: \"kubernetes.io/projected/5e99f649-b46f-4221-a022-a860fc8d998a-kube-api-access-cfw9v\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 15:05:32.226341 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.226313    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e99f649-b46f-4221-a022-a860fc8d998a-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 15:05:32.227072 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.227045    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" (UID: "467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 15:05:32.227199 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.227153    2551 scope.go:117] "RemoveContainer" containerID="0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337"
Jul 02 15:05:32.228412 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.228383    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" (UID: "467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 15:05:32.228412 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.228392    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"]
Jul 02 15:05:32.228649 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.228443    2551 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-api-access-8f7nz" (OuterVolumeSpecName: "kube-api-access-8f7nz") pod "467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" (UID: "467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c"). InnerVolumeSpecName "kube-api-access-8f7nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 15:05:32.233071 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.233052    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cb06c-predictor-bd6494ccd-g2kfz"]
Jul 02 15:05:32.234232 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.234217    2551 scope.go:117] "RemoveContainer" containerID="f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50"
Jul 02 15:05:32.234485 ip-10-0-134-54 kubenswrapper[2551]: E0702 15:05:32.234456    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50\": container with ID starting with f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50 not found: ID does not exist" containerID="f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50"
Jul 02 15:05:32.234550 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.234493    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50"} err="failed to get container status \"f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50\": rpc error: code = NotFound desc = could not find container \"f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50\": container with ID starting with f638356122786f60d30f03e80fc3fe9234aa0957e2b61e64a9ef84cedf2f2c50 not found: ID does not exist"
Jul 02 15:05:32.234550 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.234516    2551 scope.go:117] "RemoveContainer" containerID="0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337"
Jul 02 15:05:32.234723 ip-10-0-134-54 kubenswrapper[2551]: E0702 15:05:32.234705    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337\": container with ID starting with 0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337 not found: ID does not exist" containerID="0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337"
Jul 02 15:05:32.234768 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.234741    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337"} err="failed to get container status \"0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337\": rpc error: code = NotFound desc = could not find container \"0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337\": container with ID starting with 0dbc376e00b247e954aeecc0f6901fa648fa0bedcf03ca10115e327f77398337 not found: ID does not exist"
Jul 02 15:05:32.326711 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.326686    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f7nz\" (UniqueName: \"kubernetes.io/projected/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-api-access-8f7nz\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 15:05:32.326711 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.326707    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 15:05:32.326834 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.326722    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c-proxy-tls\") on node \"ip-10-0-134-54.ec2.internal\" DevicePath \"\""
Jul 02 15:05:32.529562 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.529537    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"]
Jul 02 15:05:32.534101 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:32.534079    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb06c-predictor-7df7f44c4c-5bh47"]
Jul 02 15:05:33.753148 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:33.753118    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" path="/var/lib/kubelet/pods/467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c/volumes"
Jul 02 15:05:33.753912 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:05:33.753888    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" path="/var/lib/kubelet/pods/5e99f649-b46f-4221-a022-a860fc8d998a/volumes"
Jul 02 15:06:05.128184 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:05.128105    2551 ???:1] "http: TLS handshake error from 10.0.143.132:44052: EOF"
Jul 02 15:06:05.143296 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:05.143264    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qzshm_e3e26069-93f9-43ce-9605-f1da44d37f09/global-pull-secret-syncer/0.log"
Jul 02 15:06:05.393376 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:05.393295    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wzk2v_a885d1ba-0164-40a1-a476-b982c984dbe7/konnectivity-agent/0.log"
Jul 02 15:06:05.433611 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:05.433585    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-54.ec2.internal_b9e4c4239276a54ecd642935c5ffe6cd/haproxy/0.log"
Jul 02 15:06:09.254548 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:09.254516    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-cmg89_0fbe7466-cf96-4e32-be65-042c270a0f6f/cluster-monitoring-operator/0.log"
Jul 02 15:06:09.520917 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:09.520842    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9m2n_06cdddc9-e1d5-4d17-8edc-bbc4d6936508/node-exporter/0.log"
Jul 02 15:06:09.558530 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:09.558504    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9m2n_06cdddc9-e1d5-4d17-8edc-bbc4d6936508/kube-rbac-proxy/0.log"
Jul 02 15:06:09.589287 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:09.589266    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9m2n_06cdddc9-e1d5-4d17-8edc-bbc4d6936508/init-textfile/0.log"
Jul 02 15:06:12.828958 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:12.828925    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-tnwc2_866aca15-9498-48dd-bee5-7504840402f1/download-server/0.log"
Jul 02 15:06:13.874244 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874209    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"]
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874508    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874519    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874528    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874534    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874544    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874552    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874563    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874568    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874577    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874582    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874589    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874594    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874599    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874605    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874611    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874618    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kube-rbac-proxy"
Jul 02 15:06:13.874624 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874629    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874635    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874641    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874646    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874654    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874660    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874665    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874670    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874676    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874681    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874686    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874691    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874699    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874708    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874715    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874720    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874759    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874765    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874773    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874781    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874789    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874794    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874799    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9256f2-780e-40cc-bfb8-343cbdf42f3f" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874803    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="467d3d03-2a76-4f21-af2d-7f0eb0ee6a3c" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874807    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="389a18ea-e78b-461f-a2ec-b5071b01e38f" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874812    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874817    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874822    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c220f358-6343-4525-880a-e0ebfb69e173" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874826    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e99f649-b46f-4221-a022-a860fc8d998a" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874831    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b869f464-c7f0-4c3a-9a25-29730c80d07b" containerName="kube-rbac-proxy"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874835    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b37e0d-c12b-4e34-bcb5-bf622dee8b92" containerName="kserve-container"
Jul 02 15:06:13.875126 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.874840    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee64839-23c4-465a-a0c5-e6daf981a63c" containerName="kube-rbac-proxy"
Jul 02 15:06:13.877607 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.877592    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:13.880598 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.880577    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvcf4\"/\"kube-root-ca.crt\""
Jul 02 15:06:13.881855 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.881840    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rvcf4\"/\"default-dockercfg-xczzd\""
Jul 02 15:06:13.881937 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.881858    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvcf4\"/\"openshift-service-ca.crt\""
Jul 02 15:06:13.887515 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.887496    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"]
Jul 02 15:06:13.916392 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.916364    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-proc\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:13.916489 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.916407    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-lib-modules\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:13.916489 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.916438    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-sys\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:13.916489 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.916460    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-podres\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:13.916675 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:13.916498    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89nw\" (UniqueName: \"kubernetes.io/projected/159f2928-4126-4c67-9970-b0610d4044f9-kube-api-access-p89nw\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.017045 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.017021    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-lib-modules\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.017192 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.017057    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-sys\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.017192 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.017079    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-podres\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.017192 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.017138    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89nw\" (UniqueName: \"kubernetes.io/projected/159f2928-4126-4c67-9970-b0610d4044f9-kube-api-access-p89nw\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.017192 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.017155    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-sys\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.017192 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.017174    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-podres\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.017192 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.017186    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-lib-modules\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.017419 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.017227    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-proc\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.017419 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.017294    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/159f2928-4126-4c67-9970-b0610d4044f9-proc\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.027538 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.027512    2551 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89nw\" (UniqueName: \"kubernetes.io/projected/159f2928-4126-4c67-9970-b0610d4044f9-kube-api-access-p89nw\") pod \"perf-node-gather-daemonset-z7ll8\" (UID: \"159f2928-4126-4c67-9970-b0610d4044f9\") " pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.187713 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.187638    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:14.324418 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.324393    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"]
Jul 02 15:06:14.324992 ip-10-0-134-54 kubenswrapper[2551]: W0702 15:06:14.324971    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod159f2928_4126_4c67_9970_b0610d4044f9.slice/crio-039f8e8f67cc4ef7565a46644543daec42a21e00f701bad1d610b129ade6a97d WatchSource:0}: Error finding container 039f8e8f67cc4ef7565a46644543daec42a21e00f701bad1d610b129ade6a97d: Status 404 returned error can't find the container with id 039f8e8f67cc4ef7565a46644543daec42a21e00f701bad1d610b129ade6a97d
Jul 02 15:06:14.353430 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.353402    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8" event={"ID":"159f2928-4126-4c67-9970-b0610d4044f9","Type":"ContainerStarted","Data":"039f8e8f67cc4ef7565a46644543daec42a21e00f701bad1d610b129ade6a97d"}
Jul 02 15:06:14.483769 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.483691    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kbvnk_0a8a8c4d-b364-4613-a6a1-7767c76a3efd/dns/0.log"
Jul 02 15:06:14.507810 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.507781    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kbvnk_0a8a8c4d-b364-4613-a6a1-7767c76a3efd/kube-rbac-proxy/0.log"
Jul 02 15:06:14.690785 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:14.690756    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z2vnx_9c7b1df8-a5d3-4087-9362-725f43f825d6/dns-node-resolver/0.log"
Jul 02 15:06:15.233053 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:15.233024    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4f9jp_bf27d440-b641-4efc-9a92-273c9bbef679/node-ca/0.log"
Jul 02 15:06:15.357830 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:15.357793    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8" event={"ID":"159f2928-4126-4c67-9970-b0610d4044f9","Type":"ContainerStarted","Data":"3e994bd8f357f3c69830deae404816ba393f9cd15e970bd952ddc40efc6e8a90"}
Jul 02 15:06:15.358006 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:15.357925    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:15.379369 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:15.379322    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8" podStartSLOduration=2.379306984 podStartE2EDuration="2.379306984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 15:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 15:06:15.377223029 +0000 UTC m=+2982.140608349" watchObservedRunningTime="2026-07-02 15:06:15.379306984 +0000 UTC m=+2982.142692300"
Jul 02 15:06:16.486480 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:16.486432    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s2zwr_a92336ee-ed07-4194-9236-03c1c360edbd/serve-healthcheck-canary/0.log"
Jul 02 15:06:17.014806 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:17.014760    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sh7pn_f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3/kube-rbac-proxy/0.log"
Jul 02 15:06:17.040574 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:17.040549    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sh7pn_f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3/exporter/0.log"
Jul 02 15:06:17.067235 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:17.067214    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sh7pn_f90b6ffa-cfd9-44bf-bf17-7b9da1c50ff3/extractor/0.log"
Jul 02 15:06:19.149703 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:19.149659    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7b9c77fff9-z5h9f_d0e6ea43-915a-4c51-9dcd-923b99016b90/manager/0.log"
Jul 02 15:06:19.206531 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:19.206510    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-wnhvt_9260c524-62d7-4478-8350-567dca7b65a8/server/0.log"
Jul 02 15:06:19.358631 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:19.358588    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-bl79l_5396f43d-39a7-414c-808c-52a64fc6ca1e/manager/0.log"
Jul 02 15:06:19.436065 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:19.435995    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-dfxpj_37afa26a-df0c-4a24-a157-d884de7614c8/seaweedfs/0.log"
Jul 02 15:06:21.375396 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:21.375369    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rvcf4/perf-node-gather-daemonset-z7ll8"
Jul 02 15:06:23.551034 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:23.551004    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-hcvfr_3d582354-6c0a-4ac1-a13e-32ab9f7f48df/migrator/0.log"
Jul 02 15:06:23.575175 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:23.575152    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-hcvfr_3d582354-6c0a-4ac1-a13e-32ab9f7f48df/graceful-termination/0.log"
Jul 02 15:06:25.092246 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.092217    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jn8cz_fae8e80b-baa0-4195-9657-aefc73c89147/kube-multus-additional-cni-plugins/0.log"
Jul 02 15:06:25.115531 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.115434    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jn8cz_fae8e80b-baa0-4195-9657-aefc73c89147/egress-router-binary-copy/0.log"
Jul 02 15:06:25.141793 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.141766    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jn8cz_fae8e80b-baa0-4195-9657-aefc73c89147/cni-plugins/0.log"
Jul 02 15:06:25.169960 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.169935    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jn8cz_fae8e80b-baa0-4195-9657-aefc73c89147/bond-cni-plugin/0.log"
Jul 02 15:06:25.193412 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.193371    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jn8cz_fae8e80b-baa0-4195-9657-aefc73c89147/routeoverride-cni/0.log"
Jul 02 15:06:25.216036 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.216010    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jn8cz_fae8e80b-baa0-4195-9657-aefc73c89147/whereabouts-cni-bincopy/0.log"
Jul 02 15:06:25.238946 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.238922    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jn8cz_fae8e80b-baa0-4195-9657-aefc73c89147/whereabouts-cni/0.log"
Jul 02 15:06:25.443376 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.443297    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n6zrx_bde91108-8391-455f-ae1f-ed867f2e9bdf/kube-multus/0.log"
Jul 02 15:06:25.524709 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.524678    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gr9j2_79b5c08b-176c-4374-a72b-97d488dcc046/network-metrics-daemon/0.log"
Jul 02 15:06:25.545576 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:25.545558    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gr9j2_79b5c08b-176c-4374-a72b-97d488dcc046/kube-rbac-proxy/0.log"
Jul 02 15:06:26.331543 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:26.331511    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-controller/0.log"
Jul 02 15:06:26.353394 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:26.353367    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/1.log"
Jul 02 15:06:26.353545 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:26.353429    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovn-acl-logging/0.log"
Jul 02 15:06:26.376094 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:26.376072    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/kube-rbac-proxy-node/0.log"
Jul 02 15:06:26.399668 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:26.399647    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 15:06:26.420662 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:26.420638    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/northd/0.log"
Jul 02 15:06:26.447138 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:26.447107    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/nbdb/0.log"
Jul 02 15:06:26.471792 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:26.471769    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/sbdb/0.log"
Jul 02 15:06:26.536555 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:26.536525    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6krdm_2dfb74e7-e835-4ed2-ab1d-55985f662845/ovnkube-controller/0.log"
Jul 02 15:06:28.088297 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:28.088255    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-56499946bc-dkj7s_382bbd1e-e1ed-4dc4-8e37-cdcf1df7d16c/check-endpoints/0.log"
Jul 02 15:06:28.137795 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:28.137769    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jpqgc_065c8aea-2847-43cd-bff7-ba1e706a032d/network-check-target-container/0.log"
Jul 02 15:06:29.211907 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:29.211884    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8qtb7_d29ddb9d-0592-481b-96f9-90acc879de60/iptables-alerter/0.log"
Jul 02 15:06:29.933322 ip-10-0-134-54 kubenswrapper[2551]: I0702 15:06:29.933289    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rcn6r_daa7d524-66f0-4937-83f2-8a7a0766dd9a/tuned/0.log"