Sep 17 18:27:03.503097 ip-10-0-136-130 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 18:27:03.503225 ip-10-0-136-130 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 18:27:03.503329 ip-10-0-136-130 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 18:27:03.503545 ip-10-0-136-130 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 18:27:13.525054 ip-10-0-136-130 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 18:27:13.525070 ip-10-0-136-130 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ad18d80fe37a48508160e78b6aa671a3 --
Sep 17 18:29:17.223589 ip-10-0-136-130 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:29:20.857523 ip-10-0-136-130 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:20.857523 ip-10-0-136-130 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:29:20.857523 ip-10-0-136-130 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:20.857523 ip-10-0-136-130 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:29:20.857523 ip-10-0-136-130 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:20.860004 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.859887    2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:29:20.863692 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863675    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:20.863692 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863691    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863695    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863699    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863702    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863706    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863709    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863712    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863715    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863718    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863721    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863723    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863726    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863728    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863731    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863734    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863737    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863740    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863743    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863746    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863748    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:20.863759 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863751    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863753    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863756    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863759    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863762    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863765    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863768    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863770    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863773    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863776    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863778    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863781    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863783    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863786    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863788    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863791    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863794    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863796    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863799    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863801    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:20.864241 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863804    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863806    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863809    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863811    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863814    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863816    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863820    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863824    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863827    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863830    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863833    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863836    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863839    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863841    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863845    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863847    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863850    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863853    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863855    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863858    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:20.864754 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863860    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863863    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863866    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863868    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863871    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863873    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863877    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863881    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863884    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863887    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863891    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863894    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863897    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863901    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863904    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863907    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863910    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863913    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863915    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:20.865278 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863919    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863921    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863924    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863926    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863929    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863931    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863934    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863938    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863941    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863944    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863947    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863949    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863952    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863954    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863957    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863959    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863962    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863965    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.863967    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864648    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:20.865755 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864654    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864658    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864661    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864663    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864667    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864671    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864674    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864677    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864680    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864682    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864685    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864688    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864690    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864693    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864696    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864698    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864701    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864703    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864706    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864710    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:20.866284 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864713    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864716    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864718    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864721    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864724    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864727    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864729    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864732    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864736    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864740    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864743    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864746    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864749    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864752    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864754    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864757    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864760    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864763    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864766    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864769    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:20.866805 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864772    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864775    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864777    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864780    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864782    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864785    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864787    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864790    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864792    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864794    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864797    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864801    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864804    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864807    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864809    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864812    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864815    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864817    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864820    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:20.867312 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864822    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864825    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864828    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864830    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864833    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864835    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864839    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864842    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864844    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864847    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864849    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864852    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864855    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864857    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864860    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864862    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864865    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864867    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864870    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864872    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:20.867802 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864875    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864877    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864880    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864882    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864885    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864888    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864891    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864893    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864896    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864899    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864901    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864904    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864906    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864909    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864911    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864914    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864918    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864921    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.864924    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865323    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:20.868294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865329    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865332    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865335    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865338    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865341    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865344    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865347    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865349    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865352    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865354    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865357    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865360    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865362    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865365    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865367    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865370    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865373    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865376    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865379    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865382    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:20.868804 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865385    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865387    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865392    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865396    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865399    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865402    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865405    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865408    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865411    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865413    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865416    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865419    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865421    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865424    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865427    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865429    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865432    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865434    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865437    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:20.869288 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865439    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865442    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865444    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865447    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865450    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865452    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865455    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865458    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865461    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865466    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865468    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865471    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865474    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865476    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865479    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865482    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865484    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865487    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865489    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865492    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:20.869773 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865495    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865497    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865500    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865503    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865505    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865508    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865510    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865513    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865515    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865518    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865521    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865524    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865526    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865529    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865531    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865534    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865536    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865540    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865545    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:20.870295 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865547    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865550    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865554    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865557    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865559    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865562    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865565    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865567    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865570    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865572    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865575    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865577    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865580    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865582    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865585    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865588    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865590    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865592    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865595    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.865598    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:20.870788 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865665    2552 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865708    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865717    2552 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865722    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865727    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865730    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865734    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865738    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865741    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865745    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865748    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865752    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865755    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865758    2552 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865763    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865766    2552 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865769    2552 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865772    2552 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865776    2552 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865779    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865782    2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865785    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865789    2552 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865792    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:29:20.871272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865795    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865798    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865802    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865805    2552 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865807    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865810    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865813    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865816    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865820    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865823    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865831    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865834    2552 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865837    2552 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865848    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865853    2552 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865856    2552 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865859    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865862    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865865    2552 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865873    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865876    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865879    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865885    2552 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865888    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865891    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:29:20.871860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865894    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865896    2552 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865899    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865902    2552 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865905    2552 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865909    2552 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865912    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865915    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865919    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865922    2552 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865925    2552 flags.go:64] FLAG: --help="false"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865927    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-136-130.ec2.internal"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865931    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865933    2552 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865936    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865940    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865943    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865948    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865950    2552 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865953    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865957    2552 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865960    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865963    2552 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865966    2552 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:29:20.872480 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865969    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865972    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865975    2552 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865978    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865981    2552 flags.go:64] FLAG: --lock-file=""
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865985    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865989    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865991    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865996    2552 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.865999    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866002    2552 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866005    2552 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866008    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866011    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866014    2552 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866017    2552 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866021    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866025    2552 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866028    2552 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866031    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866034    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866037    2552 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866041    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866044    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866046    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:29:20.873102 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866051    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866058    2552 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866061    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866064    2552 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866068    2552 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866071    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866076    2552 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866079    2552 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866082    2552 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866085    2552 flags.go:64] FLAG: --port="10250"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866088    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866092    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00843727d5e976747"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866095    2552 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866099    2552 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866102    2552 flags.go:64] FLAG: --register-node="true"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866105    2552 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866108    2552 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866111    2552 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866114    2552 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866117    2552 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866121    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866124    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866127    2552 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866130    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866133    2552 flags.go:64] FLAG: --runonce="false"
Sep 17 18:29:20.873775 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866136    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866139    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866143    2552 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866146    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866148    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866152    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866155    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866159    2552 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866162    2552 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866165    2552 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866169    2552 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866171    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866175    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866178    2552 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866181    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866186    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866189    2552 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866192    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866196    2552 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866199    2552 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866203    2552 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866206    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866209    2552 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866211    2552 flags.go:64] FLAG: --v="2"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866215    2552 flags.go:64] FLAG: --version="false"
Sep 17 18:29:20.874399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866219    2552 flags.go:64] FLAG: --vmodule=""
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866223    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.866226    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866348    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866352    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866356    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866359    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866362    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866365    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866367    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866370    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866373    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866375    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866378    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866383    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866387    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866390    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866394    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866397    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:20.875025 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866400    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866404    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866408    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866411    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866414    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866417    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866420    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866423    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866427    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866430    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866432    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866435    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866438    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866440    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866443    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866445    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866448    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866451    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866454    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866456    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:20.875530 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866459    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866461    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866464    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866467    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866469    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866471    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866474    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866477    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866480    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866482    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866485    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866487    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866490    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866493    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866495    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866498    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866500    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866503    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866505    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866508    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:20.876018 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866512    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866515    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866517    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866520    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866522    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866525    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866527    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866530    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866533    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866535    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866538    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866540    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866543    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866545    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866548    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866550    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866553    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866556    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866558    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866562    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:20.876718 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866565    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866567    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866570    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866573    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866575    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866578    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866581    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866583    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866586    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866589    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866591    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866594    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866598    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866600    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866603    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866605    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866608    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866610    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866613    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866615    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:20.877377 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866618    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866622    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.866625    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.867370    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.874595    2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.874609    2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.874629    2552 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.874635    2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874691    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874695    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874698    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874701    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874704    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874707    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:20.878215 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874710    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874713    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874716    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874719    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874721    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874724    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874727    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874730    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874733    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874735    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874738    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874741    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874744    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874747    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874750    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874752    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874755    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874757    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874760    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874763    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:20.878607 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874767    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874770    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874773    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874778    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874781    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874783    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874786    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874789    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874792    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874794    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874797    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874799    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874802    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874805    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874807    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874810    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874812    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874815    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874818    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874821    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:20.879100 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874823    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874826    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874828    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874831    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874834    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874836    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874839    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874842    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874844    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874847    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874849    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874852    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874854    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874857    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874860    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874863    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874866    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874868    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874871    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874873    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:20.879826 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874876    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874878    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874881    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874884    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874886    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874889    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874891    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874893    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874896    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874898    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874901    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874903    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874906    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874909    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874911    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874914    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874917    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874919    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874922    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874924    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:20.880721 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874927    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874930    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874932    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874936    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874939    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874942    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874945    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874949    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874951    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874954    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874957    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874961    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.874964    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.874969    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:20.881361 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875083    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875088    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875091    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875094    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875096    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875100    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875102    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875105    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875108    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875111    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875113    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875116    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875120    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875123    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875126    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875129    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875131    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875134    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875136    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875139    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:20.881758 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875141    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875144    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875147    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875151    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875154    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875157    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875160    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875162    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875165    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875168    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875170    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875173    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875175    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875178    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875181    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875183    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875185    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875188    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875191    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:20.882329 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875193    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875196    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875198    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875200    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875203    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875205    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875219    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875222    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875225    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875227    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875230    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875232    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875235    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875238    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875240    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875243    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875245    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875248    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875251    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:20.882821 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875254    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875256    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875259    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875262    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875265    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875269    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875273    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875276    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875279    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875281    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875284    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875287    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875290    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875293    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875295    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875298    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875322    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875326    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875329    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875331    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:20.883356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875334    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875336    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875339    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875342    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875344    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875347    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875349    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875352    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875355    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875357    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875360    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875366    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875369    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875372    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875374    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875376    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875380    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875383    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875386    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875389    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:20.883883 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:20.875391    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:20.884514 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.875396    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:20.884514 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.875505    2552 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:29:20.884514 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.881186    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:29:20.884514 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.882114    2552 server.go:1019] "Starting client certificate rotation"
Sep 17 18:29:20.884514 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.882215    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:20.884514 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.882250    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:20.884904 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.884887    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:20.886130 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.886105    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:20.892676 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.892661    2552 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:29:20.894206 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.894194    2552 log.go:25] "Validated CRI v1 image API"
Sep 17 18:29:20.894959 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.894943    2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:29:20.897191 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.897164    2552 fs.go:135] Filesystem UUIDs: map[57e16c6d-1716-46ca-bab3-5feb49e2c18b:/dev/nvme0n1p3 723efa97-73f7-40d8-b688-7223c55552f3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 18:29:20.897266 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.897189    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:29:20.901218 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.901119    2552 manager.go:216] Machine: {Timestamp:2026-09-17 18:29:20.900670114 +0000 UTC m=+2.110568257 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098936 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ae0f5c41d6d0bca61b1abe6241929 SystemUUID:ec2ae0f5-c41d-6d0b-ca61-b1abe6241929 BootID:ad18d80f-e37a-4850-8160-e78b6aa671a3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:0d:1d:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:0d:1d:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:17:66:ac:e7:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:29:20.901218 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.901217    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:29:20.901343 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.901330    2552 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:29:20.902970 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.902946    2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:29:20.903097 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.902972    2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:29:20.904143 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.904131    2552 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:29:20.904178 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.904146    2552 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:29:20.904178 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.904154    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:20.904178 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.904165    2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:20.904252 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.904232    2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:29:20.907337 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.907325    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:20.907451 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.907442    2552 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:29:20.907495 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.907486    2552 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:29:20.907526 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.907500    2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:29:20.907526 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.907515    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:29:20.907526 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.907526    2552 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:29:20.907646 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.907535    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:29:20.911500 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.908408    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:20.911580 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.911525    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:20.913698 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.913678    2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:29:20.917291 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.917274    2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:29:20.917370 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.917319    2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:29:20.918492 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918477    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:29:20.918586 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918495    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:29:20.918586 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918502    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:29:20.918586 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918507    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:29:20.918586 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918515    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:29:20.918586 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918521    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:29:20.918774 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918638    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:29:20.918774 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918645    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:29:20.918774 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918652    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:29:20.918774 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918658    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:29:20.918774 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918666    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:29:20.918774 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918674    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:29:20.918774 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918718    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:29:20.918774 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.918724    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:29:20.919597 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:20.919573    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:29:20.919838 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.919818    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:20.919971 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:20.919948    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:29:20.922571 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.922556    2552 server.go:1268] "Started kubelet"
Sep 17 18:29:20.922678 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.922624    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:29:20.922732 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.922659    2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:29:20.922732 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.922726    2552 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:29:20.923513 ip-10-0-136-130 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:29:20.924019 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.923776    2552 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:29:20.926550 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.926534    2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:29:20.927166 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:20.927140    2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:29:20.928535 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.928521    2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-130.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:29:20.937018 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.937002    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:29:20.937018 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.937009    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:20.937146 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:20.937070    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:20.937712 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.937694    2552 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:29:20.937712 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.937712    2552 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:29:20.937828 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.937698    2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:29:20.937875 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.937849    2552 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:29:20.937875 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.937857    2552 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:29:20.938582 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.938567    2552 factory.go:55] Registering systemd factory
Sep 17 18:29:20.938582 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.938583    2552 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:29:20.938805 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.938794    2552 factory.go:153] Registering CRI-O factory
Sep 17 18:29:20.938805 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.938806    2552 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:29:20.938874 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.938861    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:29:20.938906 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.938882    2552 factory.go:103] Registering Raw factory
Sep 17 18:29:20.938906 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.938893    2552 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:29:20.939378 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.939364    2552 manager.go:349] Starting recovery of all containers
Sep 17 18:29:20.942436 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.942413    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p66c5"
Sep 17 18:29:20.956093 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.956066    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p66c5"
Sep 17 18:29:20.959340 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:20.959293    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:29:20.959463 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:20.959333    2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:29:20.962457 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.962442    2552 manager.go:354] Recovery completed
Sep 17 18:29:20.966383 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.966370    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:20.968371 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.968297    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:20.968463 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.968388    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:20.968463 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.968399    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:20.968938 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.968926    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:29:20.968970 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.968939    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:29:20.968970 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.968927    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:20.968970 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.968958    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:20.969823 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.969808    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:20.969884 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.969835    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:20.969884 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.969849    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:20.970098 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.970087    2552 policy_none.go:49] "None policy: Start"
Sep 17 18:29:20.970133 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.970103    2552 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:29:20.970133 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.970113    2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:29:20.971077 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:20.971067    2552 policy_none.go:47] "Start"
Sep 17 18:29:21.005136 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.004948    2552 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:29:21.014683 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.005169    2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:29:21.014683 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.005181    2552 server.go:87] "Starting device plugin registration server"
Sep 17 18:29:21.014683 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.005474    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:29:21.014683 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.005484    2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:29:21.014683 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.005583    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:29:21.014683 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.005663    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:29:21.014683 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.005672    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:29:21.014683 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.006225    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:29:21.014683 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.006267    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.056791 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.056764    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:29:21.057953 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.057940    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:29:21.058041 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.057985    2552 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:29:21.058041 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.058008    2552 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:29:21.058118 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.058051    2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:29:21.059910 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.059885    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:21.105877 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.105853    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:21.107629 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.107581    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:21.107629 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.107615    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:21.107629 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.107629    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:21.107786 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.107658    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.118356 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.118341    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.118423 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.118364    2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-130.ec2.internal\": node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.135290 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.135265    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.158365 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.158345    2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal"]
Sep 17 18:29:21.158458 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.158420    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:21.159233 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.159218    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:21.159354 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.159256    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:21.159354 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.159278    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:21.160662 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.160648    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:21.160773 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.160761    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.160844 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.160787    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:21.161588 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.161568    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:21.161694 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.161595    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:21.161694 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.161608    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:21.161983 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.161946    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:21.161983 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.161976    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:21.162114 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.161988    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:21.163119 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.163104    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.163205 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.163127    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:21.163822 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.163807    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:21.163912 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.163831    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:21.163912 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.163840    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:21.183878 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.183864    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-130.ec2.internal\" not found" node="ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.188399 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.188385    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-130.ec2.internal\" not found" node="ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.236329 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.236291    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.240622 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.240596    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cdd6c0b420cf6fefc2f5f6abec316fb9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal\" (UID: \"cdd6c0b420cf6fefc2f5f6abec316fb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.240622 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.240625    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdd6c0b420cf6fefc2f5f6abec316fb9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal\" (UID: \"cdd6c0b420cf6fefc2f5f6abec316fb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.240776 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.240641    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ad3ebf3ceb1ffa8eb4766b2c0e01b6b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-130.ec2.internal\" (UID: \"7ad3ebf3ceb1ffa8eb4766b2c0e01b6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.336694 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.336667    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.341001 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.340988    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ad3ebf3ceb1ffa8eb4766b2c0e01b6b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-130.ec2.internal\" (UID: \"7ad3ebf3ceb1ffa8eb4766b2c0e01b6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.341069 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.341007    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cdd6c0b420cf6fefc2f5f6abec316fb9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal\" (UID: \"cdd6c0b420cf6fefc2f5f6abec316fb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.341069 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.341024    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdd6c0b420cf6fefc2f5f6abec316fb9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal\" (UID: \"cdd6c0b420cf6fefc2f5f6abec316fb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.341069 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.341058    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdd6c0b420cf6fefc2f5f6abec316fb9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal\" (UID: \"cdd6c0b420cf6fefc2f5f6abec316fb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.341162 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.341090    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ad3ebf3ceb1ffa8eb4766b2c0e01b6b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-130.ec2.internal\" (UID: \"7ad3ebf3ceb1ffa8eb4766b2c0e01b6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.341162 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.341090    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cdd6c0b420cf6fefc2f5f6abec316fb9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal\" (UID: \"cdd6c0b420cf6fefc2f5f6abec316fb9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.437194 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.437131    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.488576 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.488555    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.492349 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.492334    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:21.537826 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.537789    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.638229 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.638185    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.738648 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.738580    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.791978 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.791955    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:21.838749 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.838720    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.851948 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.851923    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:21.882455 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.882431    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:29:21.882863 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.882558    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:21.882863 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.882598    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:21.882863 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.882598    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:21.939567 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:21.939529    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-130.ec2.internal\" not found"
Sep 17 18:29:21.961077 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.961033    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:24:20 +0000 UTC" deadline="2028-06-14 06:08:13.935042156 +0000 UTC"
Sep 17 18:29:21.961077 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.961068    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15251h38m51.97397928s"
Sep 17 18:29:21.980489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:21.980465    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:22.027918 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:22.027887    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad3ebf3ceb1ffa8eb4766b2c0e01b6b.slice/crio-5ed80b6fe4e8b2a094b547d9517bf88e74085fe699d4cc9ca8ba81589a25b1d7 WatchSource:0}: Error finding container 5ed80b6fe4e8b2a094b547d9517bf88e74085fe699d4cc9ca8ba81589a25b1d7: Status 404 returned error can't find the container with id 5ed80b6fe4e8b2a094b547d9517bf88e74085fe699d4cc9ca8ba81589a25b1d7
Sep 17 18:29:22.028122 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:22.028104    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd6c0b420cf6fefc2f5f6abec316fb9.slice/crio-821d16e47b11fdcdbfabfbd1e0cea73eb8125c5e0a74d9e69d32c31a58b922cb WatchSource:0}: Error finding container 821d16e47b11fdcdbfabfbd1e0cea73eb8125c5e0a74d9e69d32c31a58b922cb: Status 404 returned error can't find the container with id 821d16e47b11fdcdbfabfbd1e0cea73eb8125c5e0a74d9e69d32c31a58b922cb
Sep 17 18:29:22.035199 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.035179    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:29:22.037955 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.037937    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:22.051984 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.051962    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:22.053249 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.053234    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal"
Sep 17 18:29:22.061610 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.061558    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal" event={"ID":"cdd6c0b420cf6fefc2f5f6abec316fb9","Type":"ContainerStarted","Data":"821d16e47b11fdcdbfabfbd1e0cea73eb8125c5e0a74d9e69d32c31a58b922cb"}
Sep 17 18:29:22.062499 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.062480    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal" event={"ID":"7ad3ebf3ceb1ffa8eb4766b2c0e01b6b","Type":"ContainerStarted","Data":"5ed80b6fe4e8b2a094b547d9517bf88e74085fe699d4cc9ca8ba81589a25b1d7"}
Sep 17 18:29:22.065075 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.065061    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:22.908635 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.908609    2552 apiserver.go:52] "Watching apiserver"
Sep 17 18:29:22.918381 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.918358    2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:29:22.918837 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.918813    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp","openshift-cluster-node-tuning-operator/tuned-zznzl","openshift-dns/node-resolver-px26v","openshift-multus/multus-6swlj","openshift-multus/network-metrics-daemon-bw2tp","openshift-network-diagnostics/network-check-target-g8m7p","kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal","openshift-image-registry/node-ca-mgf76","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal","openshift-multus/multus-additional-cni-plugins-q9glr","openshift-network-operator/iptables-alerter-gbq6n","openshift-ovn-kubernetes/ovnkube-node-tg7sn","kube-system/global-pull-secret-syncer-p45zn","kube-system/konnectivity-agent-w2dj2"]
Sep 17 18:29:22.920767 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.920745    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:22.922050 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.922022    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:22.922166 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:22.922093    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:22.924100 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.923782    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:29:22.924100 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.923782    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:29:22.924100 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.923961    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:29:22.924100 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.924065    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:29:22.924274 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.924103    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9r68\""
Sep 17 18:29:22.924274 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.924107    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:29:22.924274 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.924248    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:22.925609 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.925591    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.926717 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.926698    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:29:22.926809 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.926772    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sc5jp\""
Sep 17 18:29:22.926987 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.926972    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:29:22.929327 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.928175    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:29:22.929547 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.929524    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vgjbj\""
Sep 17 18:29:22.929726 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.929704    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:22.929835 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:22.929798    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:22.932033 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.932016    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:22.932118 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:22.932073    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:22.934479 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.934462    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:22.936587 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.936223    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:22.937361 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.937227    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:29:22.937610 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.937592    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:29:22.937696 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.937639    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bnch9\""
Sep 17 18:29:22.937696 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.937672    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:22.937858 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.937836    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.939736 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.939703    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:29:22.939853 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.939837    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:22.940568 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.940291    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:29:22.940568 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.940320    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:29:22.940568 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.940351    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b8p2k\""
Sep 17 18:29:22.940826 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.940785    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:22.940993 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.940977    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:22.941607 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.941591    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:22.942378 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.942357    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:29:22.942451 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.942360    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlgmz\""
Sep 17 18:29:22.943232 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.943212    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:29:22.943440 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.943420    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:22.943650 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.943619    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-crrgt\""
Sep 17 18:29:22.943875 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.943857    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:29:22.944269 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.944253    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:22.944371 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.944284    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:22.944904 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.944886    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mpfb8\""
Sep 17 18:29:22.944984 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.944894    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:29:22.946698 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.946640    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:29:22.947011 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.946973    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:29:22.947096 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.947037    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-crgc6\""
Sep 17 18:29:22.947187 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.947172    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:29:22.947322 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.947292    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:29:22.948194 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948177    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18-agent-certs\") pod \"konnectivity-agent-w2dj2\" (UID: \"2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18\") " pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:22.948277 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948200    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-run\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.948277 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948214    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-tmp\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.948277 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948230    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-cni-binary-copy\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.948277 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948244    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-sysctl-conf\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.948277 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948267    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-etc-selinux\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948325    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-socket-dir-parent\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948364    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948365    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-run-k8s-cni-cncf-io\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948409    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-etc-kubernetes\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948430    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-lib-modules\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948454    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948476    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba444217-b1b1-4f63-9579-d097b752acaa-hosts-file\") pod \"node-resolver-px26v\" (UID: \"ba444217-b1b1-4f63-9579-d097b752acaa\") " pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948453    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948500    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-var-lib-cni-bin\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948521    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-host\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.948559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948544    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-system-cni-dir\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948566    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8rp8\" (UniqueName: \"kubernetes.io/projected/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-kube-api-access-l8rp8\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948590    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svj4p\" (UniqueName: \"kubernetes.io/projected/69ce5590-e821-42c5-972a-d6655b7f4c75-kube-api-access-svj4p\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948630    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-hostroot\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948650    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-var-lib-kubelet\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948675    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948700    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7-serviceca\") pod \"node-ca-mgf76\" (UID: \"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7\") " pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948724    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-socket-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948761    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-device-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948782    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-modprobe-d\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948817    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9bx\" (UniqueName: \"kubernetes.io/projected/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-kube-api-access-cp9bx\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948853    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948881    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45n5s\" (UniqueName: \"kubernetes.io/projected/ba444217-b1b1-4f63-9579-d097b752acaa-kube-api-access-45n5s\") pod \"node-resolver-px26v\" (UID: \"ba444217-b1b1-4f63-9579-d097b752acaa\") " pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948919    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-var-lib-kubelet\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948951    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcxh\" (UniqueName: \"kubernetes.io/projected/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-kube-api-access-tvcxh\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948976    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18-konnectivity-ca\") pod \"konnectivity-agent-w2dj2\" (UID: \"2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18\") " pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:22.949198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.948996    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-sys\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949018    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-cnibin\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949063    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-sysconfig\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949084    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-sysctl-d\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949112    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-tuned\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949135    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949158    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-conf-dir\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949189    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7-host\") pod \"node-ca-mgf76\" (UID: \"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7\") " pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949221    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btgn\" (UniqueName: \"kubernetes.io/projected/9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7-kube-api-access-4btgn\") pod \"node-ca-mgf76\" (UID: \"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7\") " pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949244    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-systemd\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949268    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-var-lib-cni-multus\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949288    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-kubernetes\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949328    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949348    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-registration-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949361    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-run-netns\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949374    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-run-multus-certs\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.949946 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949410    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949438    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949457    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9337787-7c1b-4e8c-add0-59cd333bbd43-kubelet-config\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949486    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-sys-fs\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949507    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-system-cni-dir\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949521    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba444217-b1b1-4f63-9579-d097b752acaa-tmp-dir\") pod \"node-resolver-px26v\" (UID: \"ba444217-b1b1-4f63-9579-d097b752acaa\") " pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949543    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-daemon-config\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949564    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949587    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9337787-7c1b-4e8c-add0-59cd333bbd43-dbus\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949619    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-cni-dir\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949649    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-cnibin\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.950701 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.949666    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrf6\" (UniqueName: \"kubernetes.io/projected/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-kube-api-access-gcrf6\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:22.965168 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:22.965150    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:23.004410 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.004377    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-87rh8"
Sep 17 18:29:23.026518 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.026486    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-87rh8"
Sep 17 18:29:23.039103 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.039081    2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:29:23.049842 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.049817    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-var-lib-cni-bin\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.049957 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.049847    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-host\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.049957 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.049875    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-slash\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.049957 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.049900    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-system-cni-dir\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.049957 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.049918    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8rp8\" (UniqueName: \"kubernetes.io/projected/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-kube-api-access-l8rp8\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.049957 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.049932    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-var-lib-cni-bin\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.049957 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.049950    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-host\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.049939    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-kubelet\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050007    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-system-cni-dir\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050007    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-etc-openvswitch\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050048    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-cni-netd\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050077    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svj4p\" (UniqueName: \"kubernetes.io/projected/69ce5590-e821-42c5-972a-d6655b7f4c75-kube-api-access-svj4p\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050101    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-hostroot\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050123    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-var-lib-kubelet\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050147    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29612002-70f1-4918-9157-e130631bfdc5-host-slash\") pod \"iptables-alerter-gbq6n\" (UID: \"29612002-70f1-4918-9157-e130631bfdc5\") " pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050170    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jfb\" (UniqueName: \"kubernetes.io/projected/29612002-70f1-4918-9157-e130631bfdc5-kube-api-access-m8jfb\") pod \"iptables-alerter-gbq6n\" (UID: \"29612002-70f1-4918-9157-e130631bfdc5\") " pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050192    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-run-netns\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050195    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-hostroot\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050202    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-var-lib-kubelet\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050214    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-log-socket\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050249    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.050265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050275    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050292    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7-serviceca\") pod \"node-ca-mgf76\" (UID: \"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7\") " pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050417    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29612002-70f1-4918-9157-e130631bfdc5-iptables-alerter-script\") pod \"iptables-alerter-gbq6n\" (UID: \"29612002-70f1-4918-9157-e130631bfdc5\") " pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050450    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-socket-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050473    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-device-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050498    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-modprobe-d\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050523    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9bx\" (UniqueName: \"kubernetes.io/projected/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-kube-api-access-cp9bx\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050541    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-device-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050549    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050580    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45n5s\" (UniqueName: \"kubernetes.io/projected/ba444217-b1b1-4f63-9579-d097b752acaa-kube-api-access-45n5s\") pod \"node-resolver-px26v\" (UID: \"ba444217-b1b1-4f63-9579-d097b752acaa\") " pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050598    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050598    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-socket-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050631    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-var-lib-kubelet\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050651    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-modprobe-d\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050656    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcxh\" (UniqueName: \"kubernetes.io/projected/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-kube-api-access-tvcxh\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050775    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18-konnectivity-ca\") pod \"konnectivity-agent-w2dj2\" (UID: \"2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18\") " pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050719    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-var-lib-kubelet\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.050988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050722    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7-serviceca\") pod \"node-ca-mgf76\" (UID: \"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7\") " pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050810    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-sys\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050843    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-run-systemd\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050870    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-cnibin\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050894    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-sysconfig\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050911    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-sys\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050918    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-sysctl-d\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050935    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-cnibin\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050948    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-tuned\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050971    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-sysconfig\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050978    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.050999    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-run-openvswitch\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051016    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952bp\" (UniqueName: \"kubernetes.io/projected/bc894442-872d-4d68-9423-24d51bcc78fb-kube-api-access-952bp\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051049    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-sysctl-d\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051062    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-conf-dir\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051078    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-conf-dir\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051094    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7-host\") pod \"node-ca-mgf76\" (UID: \"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7\") " pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051119    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btgn\" (UniqueName: \"kubernetes.io/projected/9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7-kube-api-access-4btgn\") pod \"node-ca-mgf76\" (UID: \"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7\") " pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:23.051899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051137    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7-host\") pod \"node-ca-mgf76\" (UID: \"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7\") " pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051143    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-systemd\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051174    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc894442-872d-4d68-9423-24d51bcc78fb-ovn-node-metrics-cert\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051240    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-systemd\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051252    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-var-lib-cni-multus\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051258    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051296    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18-konnectivity-ca\") pod \"konnectivity-agent-w2dj2\" (UID: \"2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18\") " pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051298    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-kubernetes\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051344    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-var-lib-cni-multus\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051407    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-kubernetes\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051427    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-node-log\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051455    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051481    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc894442-872d-4d68-9423-24d51bcc78fb-env-overrides\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051524    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051549    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-registration-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051560    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051575    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-run-netns\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.052735 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051598    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-run-multus-certs\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051628    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-registration-dir\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.051634    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051666    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-run-multus-certs\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051666    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-run-netns\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051686    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.051710    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret podName:e9337787-7c1b-4e8c-add0-59cd333bbd43 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:23.551691826 +0000 UTC m=+4.761589956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret") pod "global-pull-secret-syncer-p45zn" (UID: "e9337787-7c1b-4e8c-add0-59cd333bbd43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051745    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc894442-872d-4d68-9423-24d51bcc78fb-ovnkube-script-lib\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051774    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051801    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9337787-7c1b-4e8c-add0-59cd333bbd43-kubelet-config\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-sys-fs\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051852    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-system-cni-dir\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051868    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e9337787-7c1b-4e8c-add0-59cd333bbd43-kubelet-config\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051876    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba444217-b1b1-4f63-9579-d097b752acaa-tmp-dir\") pod \"node-resolver-px26v\" (UID: \"ba444217-b1b1-4f63-9579-d097b752acaa\") " pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051892    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-sys-fs\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051913    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-daemon-config\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051936    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:23.053674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051939    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-system-cni-dir\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051960    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc894442-872d-4d68-9423-24d51bcc78fb-ovnkube-config\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.051984    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9337787-7c1b-4e8c-add0-59cd333bbd43-dbus\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052004    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-cni-dir\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052025    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-cnibin\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052051    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052046    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrf6\" (UniqueName: \"kubernetes.io/projected/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-kube-api-access-gcrf6\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052092    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18-agent-certs\") pod \"konnectivity-agent-w2dj2\" (UID: \"2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18\") " pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052135    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-run\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052154    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-tmp\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052186    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba444217-b1b1-4f63-9579-d097b752acaa-tmp-dir\") pod \"node-resolver-px26v\" (UID: \"ba444217-b1b1-4f63-9579-d097b752acaa\") " pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052197    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-systemd-units\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052213    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-cni-binary-copy\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052237    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-sysctl-conf\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052264    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-run-ovn\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.052292    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052288    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.054628 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.052361    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs podName:876396f2-26fd-4a04-a6f0-00a1e8d41c3c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:23.552346222 +0000 UTC m=+4.762244355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs") pod "network-metrics-daemon-bw2tp" (UID: "876396f2-26fd-4a04-a6f0-00a1e8d41c3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052444    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-sysctl-conf\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052448    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-daemon-config\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052290    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-cni-bin\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052498    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e9337787-7c1b-4e8c-add0-59cd333bbd43-dbus\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052510    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-etc-selinux\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052518    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-cni-dir\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052532    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-cnibin\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052536    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-socket-dir-parent\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052569    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-run-k8s-cni-cncf-io\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052579    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-multus-socket-dir-parent\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052594    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-etc-kubernetes\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052596    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/69ce5590-e821-42c5-972a-d6655b7f4c75-etc-selinux\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052618    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-host-run-k8s-cni-cncf-io\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052627    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-run\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052618    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-cni-binary-copy\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052647    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-lib-modules\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052641    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-etc-kubernetes\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.055484 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052673    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.056246 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052698    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-var-lib-openvswitch\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.056246 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052735    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba444217-b1b1-4f63-9579-d097b752acaa-hosts-file\") pod \"node-resolver-px26v\" (UID: \"ba444217-b1b1-4f63-9579-d097b752acaa\") " pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:23.056246 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052743    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-lib-modules\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.056246 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052787    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba444217-b1b1-4f63-9579-d097b752acaa-hosts-file\") pod \"node-resolver-px26v\" (UID: \"ba444217-b1b1-4f63-9579-d097b752acaa\") " pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:23.056246 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.052853    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.056246 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.055116    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-tmp\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.056246 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.055683    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-etc-tuned\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.056246 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.055790    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18-agent-certs\") pod \"konnectivity-agent-w2dj2\" (UID: \"2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18\") " pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:23.059957 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.059824    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:23.059957 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.059847    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:23.059957 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.059888    2552 projected.go:196] Error preparing data for projected volume kube-api-access-p586n for pod openshift-network-diagnostics/network-check-target-g8m7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:23.059957 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.059940    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n podName:6970b20c-83ca-4006-b799-7157e09bf405 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:23.559924494 +0000 UTC m=+4.769822637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p586n" (UniqueName: "kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n") pod "network-check-target-g8m7p" (UID: "6970b20c-83ca-4006-b799-7157e09bf405") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:23.061413 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.061394    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8rp8\" (UniqueName: \"kubernetes.io/projected/eb7fa2bb-b182-4372-ab92-0fef20c7dbab-kube-api-access-l8rp8\") pod \"tuned-zznzl\" (UID: \"eb7fa2bb-b182-4372-ab92-0fef20c7dbab\") " pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.062270 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.062251    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svj4p\" (UniqueName: \"kubernetes.io/projected/69ce5590-e821-42c5-972a-d6655b7f4c75-kube-api-access-svj4p\") pod \"aws-ebs-csi-driver-node-rllmp\" (UID: \"69ce5590-e821-42c5-972a-d6655b7f4c75\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.062842 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.062820    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9bx\" (UniqueName: \"kubernetes.io/projected/ca1f1c1d-41b5-4d8f-99cd-925be255adf0-kube-api-access-cp9bx\") pod \"multus-additional-cni-plugins-q9glr\" (UID: \"ca1f1c1d-41b5-4d8f-99cd-925be255adf0\") " pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.067806 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.067784    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btgn\" (UniqueName: \"kubernetes.io/projected/9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7-kube-api-access-4btgn\") pod \"node-ca-mgf76\" (UID: \"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7\") " pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:23.069147 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.069124    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45n5s\" (UniqueName: \"kubernetes.io/projected/ba444217-b1b1-4f63-9579-d097b752acaa-kube-api-access-45n5s\") pod \"node-resolver-px26v\" (UID: \"ba444217-b1b1-4f63-9579-d097b752acaa\") " pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:23.069719 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.069702    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrf6\" (UniqueName: \"kubernetes.io/projected/ccbcf9b6-16ae-4b48-8661-a0d06e5d204b-kube-api-access-gcrf6\") pod \"multus-6swlj\" (UID: \"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b\") " pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.070211 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.070194    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcxh\" (UniqueName: \"kubernetes.io/projected/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-kube-api-access-tvcxh\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:23.153908 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.153872    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc894442-872d-4d68-9423-24d51bcc78fb-ovnkube-config\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.153908 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.153915    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-systemd-units\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.153931    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-run-ovn\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.153946    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-cni-bin\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.153967    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-var-lib-openvswitch\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.153991    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-slash\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154016    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-run-ovn\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154017    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-kubelet\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154048    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-kubelet\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154051    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-slash\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154019    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-cni-bin\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154066    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-etc-openvswitch\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154014    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-systemd-units\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154091    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-var-lib-openvswitch\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154093    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-cni-netd\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154116    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-etc-openvswitch\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154123    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-cni-netd\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154139 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154124    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29612002-70f1-4918-9157-e130631bfdc5-host-slash\") pod \"iptables-alerter-gbq6n\" (UID: \"29612002-70f1-4918-9157-e130631bfdc5\") " pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154167    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jfb\" (UniqueName: \"kubernetes.io/projected/29612002-70f1-4918-9157-e130631bfdc5-kube-api-access-m8jfb\") pod \"iptables-alerter-gbq6n\" (UID: \"29612002-70f1-4918-9157-e130631bfdc5\") " pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154199    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-run-netns\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154216    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29612002-70f1-4918-9157-e130631bfdc5-host-slash\") pod \"iptables-alerter-gbq6n\" (UID: \"29612002-70f1-4918-9157-e130631bfdc5\") " pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154224    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-log-socket\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154256    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-log-socket\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154263    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154257    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-run-netns\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154315    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154357    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29612002-70f1-4918-9157-e130631bfdc5-iptables-alerter-script\") pod \"iptables-alerter-gbq6n\" (UID: \"29612002-70f1-4918-9157-e130631bfdc5\") " pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154389    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-run-systemd\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154414    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-run-openvswitch\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154439    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952bp\" (UniqueName: \"kubernetes.io/projected/bc894442-872d-4d68-9423-24d51bcc78fb-kube-api-access-952bp\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154458    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-run-systemd\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154461    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc894442-872d-4d68-9423-24d51bcc78fb-ovn-node-metrics-cert\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154498    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-node-log\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154461    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-run-openvswitch\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.154850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154609    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.155489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154643    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc894442-872d-4d68-9423-24d51bcc78fb-env-overrides\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.155489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154670    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bc894442-872d-4d68-9423-24d51bcc78fb-ovnkube-config\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.155489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154680    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc894442-872d-4d68-9423-24d51bcc78fb-ovnkube-script-lib\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.155489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154791    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.155489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.154858    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bc894442-872d-4d68-9423-24d51bcc78fb-node-log\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.155489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.155050    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bc894442-872d-4d68-9423-24d51bcc78fb-env-overrides\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.155489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.155056    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29612002-70f1-4918-9157-e130631bfdc5-iptables-alerter-script\") pod \"iptables-alerter-gbq6n\" (UID: \"29612002-70f1-4918-9157-e130631bfdc5\") " pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.155826 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.155805    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bc894442-872d-4d68-9423-24d51bcc78fb-ovnkube-script-lib\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.157075 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.157045    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc894442-872d-4d68-9423-24d51bcc78fb-ovn-node-metrics-cert\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.177799 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.177728    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jfb\" (UniqueName: \"kubernetes.io/projected/29612002-70f1-4918-9157-e130631bfdc5-kube-api-access-m8jfb\") pod \"iptables-alerter-gbq6n\" (UID: \"29612002-70f1-4918-9157-e130631bfdc5\") " pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.177799 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.177742    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-952bp\" (UniqueName: \"kubernetes.io/projected/bc894442-872d-4d68-9423-24d51bcc78fb-kube-api-access-952bp\") pod \"ovnkube-node-tg7sn\" (UID: \"bc894442-872d-4d68-9423-24d51bcc78fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.204611 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.204581    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:23.234095 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.234071    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9glr"
Sep 17 18:29:23.241396 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.241370    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-px26v"
Sep 17 18:29:23.249050 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.249029    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6swlj"
Sep 17 18:29:23.255132 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.255111    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:23.261727 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.261709    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mgf76"
Sep 17 18:29:23.268332 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.268318    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zznzl"
Sep 17 18:29:23.274421 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.274403    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp"
Sep 17 18:29:23.283077 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.283062    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbq6n"
Sep 17 18:29:23.288605 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.288589    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:23.557057 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.556960    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:23.557057 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.557018    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:23.557268 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.557113    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:23.557268 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.557143    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:23.557268 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.557186    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret podName:e9337787-7c1b-4e8c-add0-59cd333bbd43 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:24.557168341 +0000 UTC m=+5.767066472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret") pod "global-pull-secret-syncer-p45zn" (UID: "e9337787-7c1b-4e8c-add0-59cd333bbd43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:23.557268 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.557204    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs podName:876396f2-26fd-4a04-a6f0-00a1e8d41c3c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:24.55719634 +0000 UTC m=+5.767094469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs") pod "network-metrics-daemon-bw2tp" (UID: "876396f2-26fd-4a04-a6f0-00a1e8d41c3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:23.658320 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.658272    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:23.658469 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.658416    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:23.658469 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.658434    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:23.658469 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.658444    2552 projected.go:196] Error preparing data for projected volume kube-api-access-p586n for pod openshift-network-diagnostics/network-check-target-g8m7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:23.658612 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:23.658502    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n podName:6970b20c-83ca-4006-b799-7157e09bf405 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:24.658485547 +0000 UTC m=+5.868383697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p586n" (UniqueName: "kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n") pod "network-check-target-g8m7p" (UID: "6970b20c-83ca-4006-b799-7157e09bf405") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:23.834759 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:23.834665    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:23.863464 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:23.863444    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1f1c1d_41b5_4d8f_99cd_925be255adf0.slice/crio-af6c9b78638f672bbe3ec2b786a77ffd18c41f47c665cdd4389168830be9f6a5 WatchSource:0}: Error finding container af6c9b78638f672bbe3ec2b786a77ffd18c41f47c665cdd4389168830be9f6a5: Status 404 returned error can't find the container with id af6c9b78638f672bbe3ec2b786a77ffd18c41f47c665cdd4389168830be9f6a5
Sep 17 18:29:23.864114 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:23.864075    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba444217_b1b1_4f63_9579_d097b752acaa.slice/crio-39fc17f61a1d6731f88b640d25cf334b63647cdf60a6bad3a31aa9804c3546bc WatchSource:0}: Error finding container 39fc17f61a1d6731f88b640d25cf334b63647cdf60a6bad3a31aa9804c3546bc: Status 404 returned error can't find the container with id 39fc17f61a1d6731f88b640d25cf334b63647cdf60a6bad3a31aa9804c3546bc
Sep 17 18:29:23.866582 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:23.865848    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa7c9d7_f8fb_4442_b34b_8df28ec8d9e7.slice/crio-ab2ebac90edc305434d9e6f161636747b70204cb682a5976406517ae8e444f1b WatchSource:0}: Error finding container ab2ebac90edc305434d9e6f161636747b70204cb682a5976406517ae8e444f1b: Status 404 returned error can't find the container with id ab2ebac90edc305434d9e6f161636747b70204cb682a5976406517ae8e444f1b
Sep 17 18:29:23.867356 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:23.867334    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbcf9b6_16ae_4b48_8661_a0d06e5d204b.slice/crio-e97c3f56cf7095bd5df26761adf2acd541abf2330a9a768493c28adab26168bf WatchSource:0}: Error finding container e97c3f56cf7095bd5df26761adf2acd541abf2330a9a768493c28adab26168bf: Status 404 returned error can't find the container with id e97c3f56cf7095bd5df26761adf2acd541abf2330a9a768493c28adab26168bf
Sep 17 18:29:23.869394 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:23.869235    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7fa2bb_b182_4372_ab92_0fef20c7dbab.slice/crio-817cc19b4c33da999016962e681f961bc802b1a1b534de39fd1ce4c5fec47fd1 WatchSource:0}: Error finding container 817cc19b4c33da999016962e681f961bc802b1a1b534de39fd1ce4c5fec47fd1: Status 404 returned error can't find the container with id 817cc19b4c33da999016962e681f961bc802b1a1b534de39fd1ce4c5fec47fd1
Sep 17 18:29:23.870540 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:23.870509    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc894442_872d_4d68_9423_24d51bcc78fb.slice/crio-6f53d306047673d445cd9aa590b2009edfac012699a9a47ab8e2ac1f8530bcb5 WatchSource:0}: Error finding container 6f53d306047673d445cd9aa590b2009edfac012699a9a47ab8e2ac1f8530bcb5: Status 404 returned error can't find the container with id 6f53d306047673d445cd9aa590b2009edfac012699a9a47ab8e2ac1f8530bcb5
Sep 17 18:29:24.029058 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.029017    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:23 +0000 UTC" deadline="2028-03-29 20:44:57.34097221 +0000 UTC"
Sep 17 18:29:24.029058 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.029056    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13418h15m33.311921033s"
Sep 17 18:29:24.058479 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.058453    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:24.058615 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:24.058563    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:24.067916 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.067769    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal" event={"ID":"7ad3ebf3ceb1ffa8eb4766b2c0e01b6b","Type":"ContainerStarted","Data":"9c3eeecfcddca45ef7eca2b7241d8a9062b17e189e26a1d438b84691f89a9002"}
Sep 17 18:29:24.068973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.068949    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp" event={"ID":"69ce5590-e821-42c5-972a-d6655b7f4c75","Type":"ContainerStarted","Data":"0b1ca036c22e83b40c9de373bc120fdd2cffd55273223eb23cd91ba7d094588a"}
Sep 17 18:29:24.069944 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.069928    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbq6n" event={"ID":"29612002-70f1-4918-9157-e130631bfdc5","Type":"ContainerStarted","Data":"6041e7a06f387e6b1475016ef4eae76f8fb92292bed6e2881d150ad393916e95"}
Sep 17 18:29:24.070853 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.070833    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w2dj2" event={"ID":"2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18","Type":"ContainerStarted","Data":"c222bd51207efe0f79f19b8b0e0247c304efd17691635561f7de34a296d6a6b1"}
Sep 17 18:29:24.071810 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.071793    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zznzl" event={"ID":"eb7fa2bb-b182-4372-ab92-0fef20c7dbab","Type":"ContainerStarted","Data":"817cc19b4c33da999016962e681f961bc802b1a1b534de39fd1ce4c5fec47fd1"}
Sep 17 18:29:24.072929 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.072898    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6swlj" event={"ID":"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b","Type":"ContainerStarted","Data":"e97c3f56cf7095bd5df26761adf2acd541abf2330a9a768493c28adab26168bf"}
Sep 17 18:29:24.075936 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.075917    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-px26v" event={"ID":"ba444217-b1b1-4f63-9579-d097b752acaa","Type":"ContainerStarted","Data":"39fc17f61a1d6731f88b640d25cf334b63647cdf60a6bad3a31aa9804c3546bc"}
Sep 17 18:29:24.077400 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.077380    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerStarted","Data":"6f53d306047673d445cd9aa590b2009edfac012699a9a47ab8e2ac1f8530bcb5"}
Sep 17 18:29:24.078519 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.078484    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mgf76" event={"ID":"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7","Type":"ContainerStarted","Data":"ab2ebac90edc305434d9e6f161636747b70204cb682a5976406517ae8e444f1b"}
Sep 17 18:29:24.079522 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.079503    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9glr" event={"ID":"ca1f1c1d-41b5-4d8f-99cd-925be255adf0","Type":"ContainerStarted","Data":"af6c9b78638f672bbe3ec2b786a77ffd18c41f47c665cdd4389168830be9f6a5"}
Sep 17 18:29:24.084260 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.083370    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-130.ec2.internal" podStartSLOduration=2.083350181 podStartE2EDuration="2.083350181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:24.082359237 +0000 UTC m=+5.292257391" watchObservedRunningTime="2026-09-17 18:29:24.083350181 +0000 UTC m=+5.293248341"
Sep 17 18:29:24.158784 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.158762    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:24.567698 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.567576    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:24.567698 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.567685    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:24.567918 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:24.567808    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:24.567918 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:24.567875    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret podName:e9337787-7c1b-4e8c-add0-59cd333bbd43 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:26.567854792 +0000 UTC m=+7.777752922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret") pod "global-pull-secret-syncer-p45zn" (UID: "e9337787-7c1b-4e8c-add0-59cd333bbd43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:24.568329 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:24.568295    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:24.568405 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:24.568368    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs podName:876396f2-26fd-4a04-a6f0-00a1e8d41c3c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:26.568350725 +0000 UTC m=+7.778248855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs") pod "network-metrics-daemon-bw2tp" (UID: "876396f2-26fd-4a04-a6f0-00a1e8d41c3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:24.669196 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:24.669157    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:24.669396 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:24.669337    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:24.669396 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:24.669362    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:24.669396 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:24.669375    2552 projected.go:196] Error preparing data for projected volume kube-api-access-p586n for pod openshift-network-diagnostics/network-check-target-g8m7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:24.669568 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:24.669432    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n podName:6970b20c-83ca-4006-b799-7157e09bf405 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:26.669413528 +0000 UTC m=+7.879311674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p586n" (UniqueName: "kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n") pod "network-check-target-g8m7p" (UID: "6970b20c-83ca-4006-b799-7157e09bf405") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:25.031394 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:25.031266    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:23 +0000 UTC" deadline="2028-06-30 19:36:48.388836658 +0000 UTC"
Sep 17 18:29:25.031394 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:25.031345    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15649h7m23.35749607s"
Sep 17 18:29:25.061194 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:25.060560    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:25.061194 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:25.060683    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:25.061194 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:25.061060    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:25.061194 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:25.061142    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:25.090382 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:25.089776    2552 generic.go:356] "Generic (PLEG): container finished" podID="cdd6c0b420cf6fefc2f5f6abec316fb9" containerID="003d1c9a1fb14430752120cd123eaabe63aeb432952a62fcc637793ecc3a9404" exitCode=0
Sep 17 18:29:25.090382 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:25.090323    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal" event={"ID":"cdd6c0b420cf6fefc2f5f6abec316fb9","Type":"ContainerDied","Data":"003d1c9a1fb14430752120cd123eaabe63aeb432952a62fcc637793ecc3a9404"}
Sep 17 18:29:26.059325 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:26.059281    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:26.059807 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:26.059432    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:26.103684 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:26.103616    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal" event={"ID":"cdd6c0b420cf6fefc2f5f6abec316fb9","Type":"ContainerStarted","Data":"c43a9cc513dc2dcc9db4a169804cee0babdfe942dea3dd61d66e6d53d75be5b0"}
Sep 17 18:29:26.587868 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:26.587825    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:26.588058 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:26.587878    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:26.588058 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:26.588000    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:26.588058 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:26.588059    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs podName:876396f2-26fd-4a04-a6f0-00a1e8d41c3c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:30.588040739 +0000 UTC m=+11.797938882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs") pod "network-metrics-daemon-bw2tp" (UID: "876396f2-26fd-4a04-a6f0-00a1e8d41c3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:26.588222 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:26.588121    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:26.588222 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:26.588157    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret podName:e9337787-7c1b-4e8c-add0-59cd333bbd43 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:30.588145652 +0000 UTC m=+11.798043796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret") pod "global-pull-secret-syncer-p45zn" (UID: "e9337787-7c1b-4e8c-add0-59cd333bbd43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:26.688474 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:26.688439    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:26.688657 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:26.688608    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:26.688657 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:26.688631    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:26.688657 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:26.688643    2552 projected.go:196] Error preparing data for projected volume kube-api-access-p586n for pod openshift-network-diagnostics/network-check-target-g8m7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:26.688809 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:26.688697    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n podName:6970b20c-83ca-4006-b799-7157e09bf405 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:30.688679599 +0000 UTC m=+11.898577743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p586n" (UniqueName: "kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n") pod "network-check-target-g8m7p" (UID: "6970b20c-83ca-4006-b799-7157e09bf405") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:27.058733 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:27.058653    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:27.058894 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:27.058662    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:27.058894 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:27.058813    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:27.059009 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:27.058905    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:28.059038 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:28.059001    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:28.059486 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:28.059154    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:29.059032 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:29.059000    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:29.059032 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:29.059023    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:29.059547 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:29.059138    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:29.059547 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:29.059265    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:30.058226 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:30.058197    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:30.058436 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:30.058410    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:30.621942 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:30.621887    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:30.621942 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:30.621937    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:30.622520 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:30.622053    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:30.622520 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:30.622071    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:30.622520 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:30.622122    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret podName:e9337787-7c1b-4e8c-add0-59cd333bbd43 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:38.622101448 +0000 UTC m=+19.831999579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret") pod "global-pull-secret-syncer-p45zn" (UID: "e9337787-7c1b-4e8c-add0-59cd333bbd43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:30.622520 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:30.622146    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs podName:876396f2-26fd-4a04-a6f0-00a1e8d41c3c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:38.622133732 +0000 UTC m=+19.832031918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs") pod "network-metrics-daemon-bw2tp" (UID: "876396f2-26fd-4a04-a6f0-00a1e8d41c3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:30.722969 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:30.722938    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:30.723114 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:30.723081    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:30.723114 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:30.723098    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:30.723114 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:30.723107    2552 projected.go:196] Error preparing data for projected volume kube-api-access-p586n for pod openshift-network-diagnostics/network-check-target-g8m7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:30.723279 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:30.723154    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n podName:6970b20c-83ca-4006-b799-7157e09bf405 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:38.723137592 +0000 UTC m=+19.933035736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p586n" (UniqueName: "kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n") pod "network-check-target-g8m7p" (UID: "6970b20c-83ca-4006-b799-7157e09bf405") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:31.062015 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:31.061379    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:31.062015 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:31.061483    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:31.062015 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:31.061839    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:31.062015 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:31.061926    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:32.059013 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:32.058934    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:32.059480 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:32.059067    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:33.058819 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:33.058753    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:33.058819 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:33.058796    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:33.059048 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:33.058907    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:33.059048 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:33.059019    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:34.058819 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:34.058735    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:34.058965 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:34.058878    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:35.061843 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:35.061814    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:35.062270 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:35.061928    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:35.062270 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:35.061983    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:35.062270 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:35.062071    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:36.058841 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:36.058810    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:36.059021 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:36.058934    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:37.059115 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:37.059088    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:37.059561 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:37.059209    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:37.059561 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:37.059278    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:37.059561 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:37.059395    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:38.058436 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:38.058403    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:38.058621 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:38.058532    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:38.681109 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:38.681071    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:38.681548 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:38.681166    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:38.681548 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:38.681227    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:38.681548 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:38.681272    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:38.681548 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:38.681298    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs podName:876396f2-26fd-4a04-a6f0-00a1e8d41c3c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.68127792 +0000 UTC m=+35.891176064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs") pod "network-metrics-daemon-bw2tp" (UID: "876396f2-26fd-4a04-a6f0-00a1e8d41c3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:38.681548 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:38.681339    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret podName:e9337787-7c1b-4e8c-add0-59cd333bbd43 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.68132285 +0000 UTC m=+35.891220995 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret") pod "global-pull-secret-syncer-p45zn" (UID: "e9337787-7c1b-4e8c-add0-59cd333bbd43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:38.782002 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:38.781965    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:38.782167 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:38.782091    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:38.782167 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:38.782118    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:38.782167 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:38.782132    2552 projected.go:196] Error preparing data for projected volume kube-api-access-p586n for pod openshift-network-diagnostics/network-check-target-g8m7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:38.782289 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:38.782196    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n podName:6970b20c-83ca-4006-b799-7157e09bf405 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.782181675 +0000 UTC m=+35.992079818 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p586n" (UniqueName: "kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n") pod "network-check-target-g8m7p" (UID: "6970b20c-83ca-4006-b799-7157e09bf405") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:39.059182 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:39.058733    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:39.059182 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:39.058866    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:39.059182 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:39.058917    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:39.059182 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:39.059001    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:40.058772 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:40.058739    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:40.059162 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:40.058866    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:41.058956 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:41.058927    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:41.059252 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:41.059008    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:41.059252 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:41.059080    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:41.059252 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:41.059164    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:42.058772 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.058552    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:42.058887 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:42.058816    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:42.139834 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.139803    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6swlj" event={"ID":"ccbcf9b6-16ae-4b48-8661-a0d06e5d204b","Type":"ContainerStarted","Data":"2cdf0858ad6a55aad34765642c003dad31f01f6387ddd62187555f1c17eaef74"}
Sep 17 18:29:42.141426 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.141402    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-px26v" event={"ID":"ba444217-b1b1-4f63-9579-d097b752acaa","Type":"ContainerStarted","Data":"92a63cc889a4449b324f65a0d20994eeb6eefc4a050fb0e4d30c398f56d87a26"}
Sep 17 18:29:42.147142 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.147120    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:29:42.147525 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.147504    2552 generic.go:356] "Generic (PLEG): container finished" podID="bc894442-872d-4d68-9423-24d51bcc78fb" containerID="3da421b064839533a46d7e8b101f1292a5fd9dfb609fab7030acaf365bb85dab" exitCode=1
Sep 17 18:29:42.147616 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.147573    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerStarted","Data":"ec07bd799dc4da9fad7789049574ff8cb1cc771fb1e3949cbe3551c18c54f6b6"}
Sep 17 18:29:42.147687 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.147672    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerStarted","Data":"e2f77cb08a9a23c0c1474be502ad9b9bc0c0221c396a47d4d522d7571c72d7b9"}
Sep 17 18:29:42.147724 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.147711    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerStarted","Data":"f3411112eed1c7a5ddbe0abb72c49413c0ca2c31b84f12cfdb17e9d9a999a566"}
Sep 17 18:29:42.147757 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.147740    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerStarted","Data":"eaee807e0edabbdea9a41cefe5572153172a142996174bc6bca43881aff55396"}
Sep 17 18:29:42.147810 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.147772    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerDied","Data":"3da421b064839533a46d7e8b101f1292a5fd9dfb609fab7030acaf365bb85dab"}
Sep 17 18:29:42.147857 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.147817    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerStarted","Data":"a29a3c6094bde9e56ee4800f1d6ad5c20015b5af0119216593cb4ef883acc915"}
Sep 17 18:29:42.148951 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.148931    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mgf76" event={"ID":"9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7","Type":"ContainerStarted","Data":"064e1d261cd14629458e9a25aaf021a63cba137a8fc46f207993afb154269d17"}
Sep 17 18:29:42.150497 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.150478    2552 generic.go:356] "Generic (PLEG): container finished" podID="ca1f1c1d-41b5-4d8f-99cd-925be255adf0" containerID="021540e97eb7557d4a970b131618b3dd6bda59921030c8919bf8db26b56c6a43" exitCode=0
Sep 17 18:29:42.150599 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.150544    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9glr" event={"ID":"ca1f1c1d-41b5-4d8f-99cd-925be255adf0","Type":"ContainerDied","Data":"021540e97eb7557d4a970b131618b3dd6bda59921030c8919bf8db26b56c6a43"}
Sep 17 18:29:42.153285 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.153264    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp" event={"ID":"69ce5590-e821-42c5-972a-d6655b7f4c75","Type":"ContainerStarted","Data":"bbe839fd3d0e0c69704be4047ecf6d71e1d691f08cf6155fcc91200c377de24e"}
Sep 17 18:29:42.154828 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.154810    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w2dj2" event={"ID":"2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18","Type":"ContainerStarted","Data":"86798c139df839bae3cffdbec36c0a711aae7ad22de35a767b36025efdd9acc6"}
Sep 17 18:29:42.157048 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.156702    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zznzl" event={"ID":"eb7fa2bb-b182-4372-ab92-0fef20c7dbab","Type":"ContainerStarted","Data":"d6f6ae82048d6468d03cf21820960bf6cf874860b7f9f2608a45aa256cbb843d"}
Sep 17 18:29:42.163423 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.163381    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6swlj" podStartSLOduration=3.769863736 podStartE2EDuration="21.163369859s" totalImagesPullingTime="17.393506123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:42.162671765 +0000 UTC m=+23.372569918" watchObservedRunningTime="2026-09-17 18:29:42.163369859 +0000 UTC m=+23.373268010"
Sep 17 18:29:42.163616 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.163596    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-130.ec2.internal" podStartSLOduration=20.163590701 podStartE2EDuration="20.163590701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:26.121566981 +0000 UTC m=+7.331465134" watchObservedRunningTime="2026-09-17 18:29:42.163590701 +0000 UTC m=+23.373488861"
Sep 17 18:29:42.188643 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.188604    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zznzl" podStartSLOduration=3.838135126 podStartE2EDuration="21.188585816s" totalImagesPullingTime="17.35045069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:42.188447229 +0000 UTC m=+23.398345381" watchObservedRunningTime="2026-09-17 18:29:42.188585816 +0000 UTC m=+23.398483969"
Sep 17 18:29:42.235220 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.235171    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w2dj2" podStartSLOduration=11.313489102 podStartE2EDuration="21.235160515s" totalImagesPullingTime="9.921671413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:42.234787425 +0000 UTC m=+23.444685579" watchObservedRunningTime="2026-09-17 18:29:42.235160515 +0000 UTC m=+23.445058664"
Sep 17 18:29:42.264684 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.264629    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mgf76" podStartSLOduration=3.911661823 podStartE2EDuration="21.264610584s" totalImagesPullingTime="17.352948761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:42.264455821 +0000 UTC m=+23.474353972" watchObservedRunningTime="2026-09-17 18:29:42.264610584 +0000 UTC m=+23.474508738"
Sep 17 18:29:42.320527 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.320453    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-px26v" podStartSLOduration=3.966101615 podStartE2EDuration="21.320438925s" totalImagesPullingTime="17.35433731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:42.320265192 +0000 UTC m=+23.530163345" watchObservedRunningTime="2026-09-17 18:29:42.320438925 +0000 UTC m=+23.530337078"
Sep 17 18:29:42.726943 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:42.726873    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:29:43.018343 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.018158    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:29:42.726893924Z","UUID":"721253c6-3053-4cb4-b7f8-c617616dd2e7","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:29:43.019687 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.019666    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:29:43.019795 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.019701    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:29:43.058734 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.058706    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:43.058900 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.058752    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:43.058900 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:43.058811    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:43.058900 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:43.058881    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:43.161653 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.161607    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp" event={"ID":"69ce5590-e821-42c5-972a-d6655b7f4c75","Type":"ContainerStarted","Data":"4e6633ab61ad2abb19aac10df3f4f3b96de7b9836d8b9c1ab6383fb2cfac2067"}
Sep 17 18:29:43.163550 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.163522    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbq6n" event={"ID":"29612002-70f1-4918-9157-e130631bfdc5","Type":"ContainerStarted","Data":"b0c8ed4f20c82400587a195c9076b2e806961585611dd5aa3d3050df812ab092"}
Sep 17 18:29:43.203631 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.203570    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gbq6n" podStartSLOduration=4.8602768659999995 podStartE2EDuration="22.203547964s" totalImagesPullingTime="17.343271098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:43.20235447 +0000 UTC m=+24.412252623" watchObservedRunningTime="2026-09-17 18:29:43.203547964 +0000 UTC m=+24.413446119"
Sep 17 18:29:43.479197 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.479112    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:43.479737 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:43.479713    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:44.058429 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:44.058357    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:44.058574 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:44.058461    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:44.168037 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:44.168010    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp" event={"ID":"69ce5590-e821-42c5-972a-d6655b7f4c75","Type":"ContainerStarted","Data":"caf8662c979fecb1fbef12c544ca098a1b8391ebc912b2fbd4c0eed4560efc88"}
Sep 17 18:29:44.168664 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:44.168646    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:44.168771 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:44.168714    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w2dj2"
Sep 17 18:29:44.186854 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:44.186817    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rllmp" podStartSLOduration=3.745165552 podStartE2EDuration="23.186802236s" totalImagesPullingTime="19.441636684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:44.1863942 +0000 UTC m=+25.396292353" watchObservedRunningTime="2026-09-17 18:29:44.186802236 +0000 UTC m=+25.396700388"
Sep 17 18:29:45.059046 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:45.059009    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:45.059226 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:45.059009    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:45.059226 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:45.059141    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:45.059226 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:45.059199    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:45.173512 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:45.173476    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:29:45.173948 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:45.173928    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerStarted","Data":"04fe08f7bc0829f861b255c8e91380e16e58e164855c892767b36931c3e265ff"}
Sep 17 18:29:46.059129 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:46.059097    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:46.059299 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:46.059206    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:47.058857 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:47.058567    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:47.059539 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:47.058595    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:47.059539 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:47.058874    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:47.059539 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:47.058958    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:47.182100 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:47.182075    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:29:47.182521 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:47.182485    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerStarted","Data":"fac84b41df5a7a1e38f8cc825e10e8ee95394c19a1d12eb13a0333796595dbc7"}
Sep 17 18:29:47.182744 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:47.182727    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:47.182903 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:47.182887    2552 scope.go:117] "RemoveContainer" containerID="3da421b064839533a46d7e8b101f1292a5fd9dfb609fab7030acaf365bb85dab"
Sep 17 18:29:47.184391 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:47.184368    2552 generic.go:356] "Generic (PLEG): container finished" podID="ca1f1c1d-41b5-4d8f-99cd-925be255adf0" containerID="94722e3f2cc55d8d279c2f6d2527b9ba816a95aa094fd56e7e7ef2a330369bf8" exitCode=0
Sep 17 18:29:47.184466 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:47.184400    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9glr" event={"ID":"ca1f1c1d-41b5-4d8f-99cd-925be255adf0","Type":"ContainerDied","Data":"94722e3f2cc55d8d279c2f6d2527b9ba816a95aa094fd56e7e7ef2a330369bf8"}
Sep 17 18:29:47.197980 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:47.197929    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:48.058466 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.058320    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:48.058602 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:48.058551    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:48.190797 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.190771    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:29:48.191210 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.191190    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" event={"ID":"bc894442-872d-4d68-9423-24d51bcc78fb","Type":"ContainerStarted","Data":"57e3303cbbeecb8da8e063d54c76e3ce21dc85cb80beb6c1f62469393e2b22bd"}
Sep 17 18:29:48.191640 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.191615    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:48.191750 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.191688    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:48.208161 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.208138    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:29:48.221869 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.221818    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn" podStartSLOduration=9.854196773 podStartE2EDuration="27.221799862s" totalImagesPullingTime="17.367603089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:48.219662445 +0000 UTC m=+29.429560596" watchObservedRunningTime="2026-09-17 18:29:48.221799862 +0000 UTC m=+29.431698015"
Sep 17 18:29:48.415568 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.415538    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8m7p"]
Sep 17 18:29:48.415699 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.415674    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:48.415799 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:48.415776    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:48.416239 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.416223    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p45zn"]
Sep 17 18:29:48.416333 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.416319    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:48.416409 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:48.416392    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:48.418802 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.418781    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bw2tp"]
Sep 17 18:29:48.418915 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:48.418879    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:48.419044 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:48.419020    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:49.195164 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:49.195129    2552 generic.go:356] "Generic (PLEG): container finished" podID="ca1f1c1d-41b5-4d8f-99cd-925be255adf0" containerID="cf273b13ac9dc3887be0303b7ac960b3fb41ac9f440795a21498263797e758e2" exitCode=0
Sep 17 18:29:49.195655 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:49.195211    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9glr" event={"ID":"ca1f1c1d-41b5-4d8f-99cd-925be255adf0","Type":"ContainerDied","Data":"cf273b13ac9dc3887be0303b7ac960b3fb41ac9f440795a21498263797e758e2"}
Sep 17 18:29:50.058293 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:50.058259    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:50.058293 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:50.058296    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:50.058563 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:50.058322    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:50.058563 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:50.058404    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:50.058563 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:50.058488    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:50.058719 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:50.058576    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:51.201879 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:51.201846    2552 generic.go:356] "Generic (PLEG): container finished" podID="ca1f1c1d-41b5-4d8f-99cd-925be255adf0" containerID="b6fbebbf45ed8a31176e5d0463571bb05f58a0ce6a24a9982884ae3a823dbdc3" exitCode=0
Sep 17 18:29:51.202336 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:51.201897    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9glr" event={"ID":"ca1f1c1d-41b5-4d8f-99cd-925be255adf0","Type":"ContainerDied","Data":"b6fbebbf45ed8a31176e5d0463571bb05f58a0ce6a24a9982884ae3a823dbdc3"}
Sep 17 18:29:52.058798 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:52.058761    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:52.058980 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:52.058761    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:52.058980 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:52.058902    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:52.058980 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:52.058761    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:52.058980 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:52.058958    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:52.059223 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:52.059035    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:54.059214 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.058974    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:54.059743 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.058983    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:54.059743 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.059336    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g8m7p" podUID="6970b20c-83ca-4006-b799-7157e09bf405"
Sep 17 18:29:54.059743 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.059002    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:54.059743 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.059398    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p45zn" podUID="e9337787-7c1b-4e8c-add0-59cd333bbd43"
Sep 17 18:29:54.059743 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.059528    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2tp" podUID="876396f2-26fd-4a04-a6f0-00a1e8d41c3c"
Sep 17 18:29:54.125577 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.125545    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-130.ec2.internal" event="NodeReady"
Sep 17 18:29:54.125724 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.125674    2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:29:54.179144 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.179107    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rgs4g"]
Sep 17 18:29:54.207331 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.207228    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lfc2n"]
Sep 17 18:29:54.207508 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.207362    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.210482 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.210363    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:29:54.210482 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.210386    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpkjn\""
Sep 17 18:29:54.210482 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.210446    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:29:54.229227 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.229201    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b697ddd8-559wg"]
Sep 17 18:29:54.229386 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.229352    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfc2n"
Sep 17 18:29:54.233949 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.233855    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jr68f\""
Sep 17 18:29:54.234344 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.234318    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:29:54.234483 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.234367    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:29:54.234906 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.234885    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:29:54.248147 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.248124    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfc2n"]
Sep 17 18:29:54.248286 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.248158    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rgs4g"]
Sep 17 18:29:54.248286 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.248183    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b697ddd8-559wg"]
Sep 17 18:29:54.248286 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.248274    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.257387 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.257361    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-26p5d\""
Sep 17 18:29:54.257387 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.257377    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:29:54.257522 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.257403    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:29:54.257522 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.257406    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:29:54.264104 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.264078    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:29:54.288770 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.288742    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f199ee-cba0-43b7-867f-8503480c9ff8-tmp-dir\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.288920 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.288789    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2g8\" (UniqueName: \"kubernetes.io/projected/beecf21c-b666-4309-b7af-0165216fff83-kube-api-access-fp2g8\") pod \"ingress-canary-lfc2n\" (UID: \"beecf21c-b666-4309-b7af-0165216fff83\") " pod="openshift-ingress-canary/ingress-canary-lfc2n"
Sep 17 18:29:54.288920 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.288811    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.289018 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.288921    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f199ee-cba0-43b7-867f-8503480c9ff8-config-volume\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.289018 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.288962    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrb9x\" (UniqueName: \"kubernetes.io/projected/a8f199ee-cba0-43b7-867f-8503480c9ff8-kube-api-access-zrb9x\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.289018 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.288997    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/beecf21c-b666-4309-b7af-0165216fff83-cert\") pod \"ingress-canary-lfc2n\" (UID: \"beecf21c-b666-4309-b7af-0165216fff83\") " pod="openshift-ingress-canary/ingress-canary-lfc2n"
Sep 17 18:29:54.390240 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390205    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f199ee-cba0-43b7-867f-8503480c9ff8-tmp-dir\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.390240 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390245    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4ab780-ba57-4714-b6a1-5a75369cb415-ca-trust-extracted\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.390499 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390269    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2g8\" (UniqueName: \"kubernetes.io/projected/beecf21c-b666-4309-b7af-0165216fff83-kube-api-access-fp2g8\") pod \"ingress-canary-lfc2n\" (UID: \"beecf21c-b666-4309-b7af-0165216fff83\") " pod="openshift-ingress-canary/ingress-canary-lfc2n"
Sep 17 18:29:54.390499 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390316    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.390499 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390349    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s9qx\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-kube-api-access-5s9qx\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.390499 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390378    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-installation-pull-secrets\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.390499 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390415    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f199ee-cba0-43b7-867f-8503480c9ff8-config-volume\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.390499 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390439    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-image-registry-private-configuration\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.390499 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390466    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-certificates\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.390499 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.390468    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:54.390857 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.390534    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls podName:a8f199ee-cba0-43b7-867f-8503480c9ff8 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.890520912 +0000 UTC m=+36.100419042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls") pod "dns-default-rgs4g" (UID: "a8f199ee-cba0-43b7-867f-8503480c9ff8") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:54.390857 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390718    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrb9x\" (UniqueName: \"kubernetes.io/projected/a8f199ee-cba0-43b7-867f-8503480c9ff8-kube-api-access-zrb9x\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.390857 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390767    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.390857 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390794    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-trusted-ca\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.390857 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390817    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-bound-sa-token\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.390857 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390851    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/beecf21c-b666-4309-b7af-0165216fff83-cert\") pod \"ingress-canary-lfc2n\" (UID: \"beecf21c-b666-4309-b7af-0165216fff83\") " pod="openshift-ingress-canary/ingress-canary-lfc2n"
Sep 17 18:29:54.391126 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.390943    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f199ee-cba0-43b7-867f-8503480c9ff8-tmp-dir\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.391337 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.391320    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f199ee-cba0-43b7-867f-8503480c9ff8-config-volume\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.395604 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.395583    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/beecf21c-b666-4309-b7af-0165216fff83-cert\") pod \"ingress-canary-lfc2n\" (UID: \"beecf21c-b666-4309-b7af-0165216fff83\") " pod="openshift-ingress-canary/ingress-canary-lfc2n"
Sep 17 18:29:54.400093 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.400070    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrb9x\" (UniqueName: \"kubernetes.io/projected/a8f199ee-cba0-43b7-867f-8503480c9ff8-kube-api-access-zrb9x\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.401500 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.401476    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2g8\" (UniqueName: \"kubernetes.io/projected/beecf21c-b666-4309-b7af-0165216fff83-kube-api-access-fp2g8\") pod \"ingress-canary-lfc2n\" (UID: \"beecf21c-b666-4309-b7af-0165216fff83\") " pod="openshift-ingress-canary/ingress-canary-lfc2n"
Sep 17 18:29:54.491766 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.491682    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s9qx\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-kube-api-access-5s9qx\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.491766 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.491729    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-installation-pull-secrets\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.491766 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.491761    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-image-registry-private-configuration\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.492029 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.491780    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-certificates\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.492029 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.491892    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.492029 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.491926    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-trusted-ca\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.492029 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.491949    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-bound-sa-token\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.492029 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.492004    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4ab780-ba57-4714-b6a1-5a75369cb415-ca-trust-extracted\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.492029 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.492018    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:54.492372 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.492036    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b697ddd8-559wg: secret "image-registry-tls" not found
Sep 17 18:29:54.492372 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.492091    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls podName:9d4ab780-ba57-4714-b6a1-5a75369cb415 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:54.992071547 +0000 UTC m=+36.201969694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls") pod "image-registry-6b697ddd8-559wg" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415") : secret "image-registry-tls" not found
Sep 17 18:29:54.492497 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.492426    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-certificates\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.492497 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.492484    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4ab780-ba57-4714-b6a1-5a75369cb415-ca-trust-extracted\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.493239 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.493211    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-trusted-ca\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.494611 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.494587    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-image-registry-private-configuration\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.494611 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.494608    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-installation-pull-secrets\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.502616 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.502597    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s9qx\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-kube-api-access-5s9qx\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.504959 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.504938    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-bound-sa-token\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.540088 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.540061    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfc2n"
Sep 17 18:29:54.694253 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.694218    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:54.694253 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.694260    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:54.694486 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.694413    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:54.694542 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.694487    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:54.694542 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.694488    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret podName:e9337787-7c1b-4e8c-add0-59cd333bbd43 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:26.694470042 +0000 UTC m=+67.904368193 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret") pod "global-pull-secret-syncer-p45zn" (UID: "e9337787-7c1b-4e8c-add0-59cd333bbd43") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:54.694653 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.694546    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs podName:876396f2-26fd-4a04-a6f0-00a1e8d41c3c nodeName:}" failed. No retries permitted until 2026-09-17 18:30:26.694529398 +0000 UTC m=+67.904427531 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs") pod "network-metrics-daemon-bw2tp" (UID: "876396f2-26fd-4a04-a6f0-00a1e8d41c3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:54.795499 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.795418    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:54.795679 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.795605    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:54.795679 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.795628    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:54.795679 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.795638    2552 projected.go:196] Error preparing data for projected volume kube-api-access-p586n for pod openshift-network-diagnostics/network-check-target-g8m7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:54.795828 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.795700    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n podName:6970b20c-83ca-4006-b799-7157e09bf405 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:26.795681297 +0000 UTC m=+68.005579446 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p586n" (UniqueName: "kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n") pod "network-check-target-g8m7p" (UID: "6970b20c-83ca-4006-b799-7157e09bf405") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:54.896787 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.896749    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:54.896980 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.896913    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:54.897038 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.896985    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls podName:a8f199ee-cba0-43b7-867f-8503480c9ff8 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:55.896964018 +0000 UTC m=+37.106862150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls") pod "dns-default-rgs4g" (UID: "a8f199ee-cba0-43b7-867f-8503480c9ff8") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:54.997611 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:54.997570    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:54.997837 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.997728    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:54.997837 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.997752    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b697ddd8-559wg: secret "image-registry-tls" not found
Sep 17 18:29:54.997837 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:54.997809    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls podName:9d4ab780-ba57-4714-b6a1-5a75369cb415 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:55.997789847 +0000 UTC m=+37.207687998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls") pod "image-registry-6b697ddd8-559wg" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415") : secret "image-registry-tls" not found
Sep 17 18:29:55.466456 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:55.466384    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfc2n"]
Sep 17 18:29:55.469242 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:29:55.469214    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeecf21c_b666_4309_b7af_0165216fff83.slice/crio-9e3e44e7831c5355fecf16e279d3480bafde34d769cac0bc3e4b946e6edd0a1a WatchSource:0}: Error finding container 9e3e44e7831c5355fecf16e279d3480bafde34d769cac0bc3e4b946e6edd0a1a: Status 404 returned error can't find the container with id 9e3e44e7831c5355fecf16e279d3480bafde34d769cac0bc3e4b946e6edd0a1a
Sep 17 18:29:55.903855 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:55.903817    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:55.904056 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:55.903998    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:55.904123 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:55.904075    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls podName:a8f199ee-cba0-43b7-867f-8503480c9ff8 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:57.904052466 +0000 UTC m=+39.113950609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls") pod "dns-default-rgs4g" (UID: "a8f199ee-cba0-43b7-867f-8503480c9ff8") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:56.005231 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.005148    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:56.005421 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:56.005360    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:56.005421 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:56.005381    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b697ddd8-559wg: secret "image-registry-tls" not found
Sep 17 18:29:56.005538 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:56.005447    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls podName:9d4ab780-ba57-4714-b6a1-5a75369cb415 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:58.005427727 +0000 UTC m=+39.215325871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls") pod "image-registry-6b697ddd8-559wg" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415") : secret "image-registry-tls" not found
Sep 17 18:29:56.058489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.058456    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:29:56.058489 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.058480    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:29:56.058725 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.058566    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:29:56.061273 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.061246    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:29:56.061416 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.061286    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:29:56.062581 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.062564    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rcjbl\""
Sep 17 18:29:56.062698 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.062609    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:29:56.062698 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.062623    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:29:56.062698 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.062611    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dzhw6\""
Sep 17 18:29:56.218942 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.218903    2552 generic.go:356] "Generic (PLEG): container finished" podID="ca1f1c1d-41b5-4d8f-99cd-925be255adf0" containerID="8e3e86163874cd628c5b5502b17fe80b63052fba87abed5ef44179c65617e78a" exitCode=0
Sep 17 18:29:56.219170 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.218982    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9glr" event={"ID":"ca1f1c1d-41b5-4d8f-99cd-925be255adf0","Type":"ContainerDied","Data":"8e3e86163874cd628c5b5502b17fe80b63052fba87abed5ef44179c65617e78a"}
Sep 17 18:29:56.220395 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:56.220369    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfc2n" event={"ID":"beecf21c-b666-4309-b7af-0165216fff83","Type":"ContainerStarted","Data":"9e3e44e7831c5355fecf16e279d3480bafde34d769cac0bc3e4b946e6edd0a1a"}
Sep 17 18:29:57.226413 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:57.226161    2552 generic.go:356] "Generic (PLEG): container finished" podID="ca1f1c1d-41b5-4d8f-99cd-925be255adf0" containerID="285780bb7b313926f7549fee29a9dc27bdb77c17096c33a7b60dd5fbe1842fe8" exitCode=0
Sep 17 18:29:57.227051 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:57.226195    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9glr" event={"ID":"ca1f1c1d-41b5-4d8f-99cd-925be255adf0","Type":"ContainerDied","Data":"285780bb7b313926f7549fee29a9dc27bdb77c17096c33a7b60dd5fbe1842fe8"}
Sep 17 18:29:57.919705 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:57.919623    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:29:57.919843 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:57.919816    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:57.919915 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:57.919902    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls podName:a8f199ee-cba0-43b7-867f-8503480c9ff8 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:01.919880321 +0000 UTC m=+43.129778474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls") pod "dns-default-rgs4g" (UID: "a8f199ee-cba0-43b7-867f-8503480c9ff8") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:58.020612 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:58.020578    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:29:58.020743 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:58.020719    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:58.020743 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:58.020736    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b697ddd8-559wg: secret "image-registry-tls" not found
Sep 17 18:29:58.020816 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:29:58.020789    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls podName:9d4ab780-ba57-4714-b6a1-5a75369cb415 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:02.020773326 +0000 UTC m=+43.230671461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls") pod "image-registry-6b697ddd8-559wg" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415") : secret "image-registry-tls" not found
Sep 17 18:29:58.235075 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:58.235004    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9glr" event={"ID":"ca1f1c1d-41b5-4d8f-99cd-925be255adf0","Type":"ContainerStarted","Data":"557aec2d1a1def795ef880f0b09430999eae04a65b50ae2c4ec797ef4e9b807e"}
Sep 17 18:29:58.236384 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:58.236363    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfc2n" event={"ID":"beecf21c-b666-4309-b7af-0165216fff83","Type":"ContainerStarted","Data":"d0e6ee9aeae61c4b4e78beb6aa99969265a1287a967b98384e39566071485d57"}
Sep 17 18:29:58.261399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:58.261360    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q9glr" podStartSLOduration=8.932197028000001 podStartE2EDuration="37.261347297s" totalImagesPullingTime="28.329150269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:58.259421188 +0000 UTC m=+39.469319341" watchObservedRunningTime="2026-09-17 18:29:58.261347297 +0000 UTC m=+39.471245449"
Sep 17 18:29:58.277694 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:29:58.277658    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lfc2n" podStartSLOduration=2.006702316 podStartE2EDuration="4.277644942s" totalImagesPullingTime="2.270942626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:58.277226809 +0000 UTC m=+39.487124962" watchObservedRunningTime="2026-09-17 18:29:58.277644942 +0000 UTC m=+39.487543084"
Sep 17 18:30:01.946754 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:01.946722    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:30:01.947103 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:01.946863    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:01.947103 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:01.946929    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls podName:a8f199ee-cba0-43b7-867f-8503480c9ff8 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.946915807 +0000 UTC m=+51.156813951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls") pod "dns-default-rgs4g" (UID: "a8f199ee-cba0-43b7-867f-8503480c9ff8") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:02.047997 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.047962    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:30:02.048149 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:02.048082    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:02.048149 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:02.048094    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b697ddd8-559wg: secret "image-registry-tls" not found
Sep 17 18:30:02.048149 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:02.048150    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls podName:9d4ab780-ba57-4714-b6a1-5a75369cb415 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.048136411 +0000 UTC m=+51.258034541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls") pod "image-registry-6b697ddd8-559wg" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415") : secret "image-registry-tls" not found
Sep 17 18:30:02.079037 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.076908    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq"]
Sep 17 18:30:02.080535 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.080515    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq"
Sep 17 18:30:02.083436 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.083420    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-59vsb\""
Sep 17 18:30:02.086286 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.086265    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 18:30:02.086409 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.086265    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 18:30:02.099763 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.099743    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq"]
Sep 17 18:30:02.148884 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.148858    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlj8r\" (UniqueName: \"kubernetes.io/projected/260fbd36-e31c-48d5-bf5a-71f7b1d495a3-kube-api-access-xlj8r\") pod \"migrator-76794cbd6c-pppzq\" (UID: \"260fbd36-e31c-48d5-bf5a-71f7b1d495a3\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq"
Sep 17 18:30:02.250034 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.249975    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlj8r\" (UniqueName: \"kubernetes.io/projected/260fbd36-e31c-48d5-bf5a-71f7b1d495a3-kube-api-access-xlj8r\") pod \"migrator-76794cbd6c-pppzq\" (UID: \"260fbd36-e31c-48d5-bf5a-71f7b1d495a3\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq"
Sep 17 18:30:02.262743 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.262724    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlj8r\" (UniqueName: \"kubernetes.io/projected/260fbd36-e31c-48d5-bf5a-71f7b1d495a3-kube-api-access-xlj8r\") pod \"migrator-76794cbd6c-pppzq\" (UID: \"260fbd36-e31c-48d5-bf5a-71f7b1d495a3\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq"
Sep 17 18:30:02.388607 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.388564    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq"
Sep 17 18:30:02.668321 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.668278    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq"]
Sep 17 18:30:02.670940 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:02.670916    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260fbd36_e31c_48d5_bf5a_71f7b1d495a3.slice/crio-15ec3630f1f73c74dd63f9d67e413c4e1c536a3d14c26a8336ff49e42b4dbec5 WatchSource:0}: Error finding container 15ec3630f1f73c74dd63f9d67e413c4e1c536a3d14c26a8336ff49e42b4dbec5: Status 404 returned error can't find the container with id 15ec3630f1f73c74dd63f9d67e413c4e1c536a3d14c26a8336ff49e42b4dbec5
Sep 17 18:30:02.825374 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.825348    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4tjv2"]
Sep 17 18:30:02.827806 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.827792    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:02.830992 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.830973    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:30:02.831563 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.831540    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:30:02.832616 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.832601    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:30:02.832616 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.832612    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6gvz8\""
Sep 17 18:30:02.832759 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.832646    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:30:02.846771 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.846753    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4tjv2"]
Sep 17 18:30:02.955928 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.955871    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17c175cf-5b8a-4201-bd88-88f28de96ec6-crio-socket\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:02.955928 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.955899    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r648x\" (UniqueName: \"kubernetes.io/projected/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-api-access-r648x\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:02.956244 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.955933    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:02.956244 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.955988    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:02.956244 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:02.956065    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17c175cf-5b8a-4201-bd88-88f28de96ec6-data-volume\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.057094 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.057061    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17c175cf-5b8a-4201-bd88-88f28de96ec6-crio-socket\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.057094 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.057097    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r648x\" (UniqueName: \"kubernetes.io/projected/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-api-access-r648x\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.057342 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.057129    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.057342 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.057160    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.057342 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:03.057227    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:03.057342 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:03.057236    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:03.057342 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.057293    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17c175cf-5b8a-4201-bd88-88f28de96ec6-crio-socket\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.057342 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:03.057325    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm podName:17c175cf-5b8a-4201-bd88-88f28de96ec6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.557288272 +0000 UTC m=+44.767186424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4tjv2" (UID: "17c175cf-5b8a-4201-bd88-88f28de96ec6") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:03.057597 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:03.057351    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls podName:17c175cf-5b8a-4201-bd88-88f28de96ec6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.557340826 +0000 UTC m=+44.767238957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tjv2" (UID: "17c175cf-5b8a-4201-bd88-88f28de96ec6") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:03.057597 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.057400    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17c175cf-5b8a-4201-bd88-88f28de96ec6-data-volume\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.057664 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.057611    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17c175cf-5b8a-4201-bd88-88f28de96ec6-data-volume\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.068239 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.068219    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r648x\" (UniqueName: \"kubernetes.io/projected/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-api-access-r648x\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.249251 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.249178    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq" event={"ID":"260fbd36-e31c-48d5-bf5a-71f7b1d495a3","Type":"ContainerStarted","Data":"15ec3630f1f73c74dd63f9d67e413c4e1c536a3d14c26a8336ff49e42b4dbec5"}
Sep 17 18:30:03.561552 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.561475    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.561552 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:03.561529    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:03.561774 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:03.561632    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:03.561774 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:03.561664    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:03.561774 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:03.561727    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls podName:17c175cf-5b8a-4201-bd88-88f28de96ec6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:04.561700204 +0000 UTC m=+45.771598343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tjv2" (UID: "17c175cf-5b8a-4201-bd88-88f28de96ec6") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:03.561774 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:03.561743    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm podName:17c175cf-5b8a-4201-bd88-88f28de96ec6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:04.561736074 +0000 UTC m=+45.771634204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4tjv2" (UID: "17c175cf-5b8a-4201-bd88-88f28de96ec6") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:04.253733 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.253700    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq" event={"ID":"260fbd36-e31c-48d5-bf5a-71f7b1d495a3","Type":"ContainerStarted","Data":"37e35edf0f309d8a9a5d418d8b87a1abfc3405088d90988a658e2f9a2d88f7ec"}
Sep 17 18:30:04.254055 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.253755    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq" event={"ID":"260fbd36-e31c-48d5-bf5a-71f7b1d495a3","Type":"ContainerStarted","Data":"c0372647009275c9303fced816b11c07a9657fa8e4bfea60515bf9d41d208012"}
Sep 17 18:30:04.273183 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.273145    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-pppzq" podStartSLOduration=0.900877926 podStartE2EDuration="2.273129145s" totalImagesPullingTime="1.372251219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:04.272094209 +0000 UTC m=+45.481992372" watchObservedRunningTime="2026-09-17 18:30:04.273129145 +0000 UTC m=+45.483027280"
Sep 17 18:30:04.569755 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.569725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:04.569914 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:04.569855    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:04.569914 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.569877    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:04.569914 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:04.569906    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls podName:17c175cf-5b8a-4201-bd88-88f28de96ec6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:06.569892859 +0000 UTC m=+47.779791006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tjv2" (UID: "17c175cf-5b8a-4201-bd88-88f28de96ec6") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:04.570056 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:04.569932    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:04.570056 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:04.569980    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm podName:17c175cf-5b8a-4201-bd88-88f28de96ec6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:06.569970211 +0000 UTC m=+47.779868360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4tjv2" (UID: "17c175cf-5b8a-4201-bd88-88f28de96ec6") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:04.779114 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.779080    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-8q5gx"]
Sep 17 18:30:04.780961 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.780941    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:04.784338 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.784320    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jm2lb\""
Sep 17 18:30:04.784448 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.784401    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 18:30:04.785402 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.785386    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 18:30:04.785455 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.785442    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 18:30:04.785669 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.785653    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 18:30:04.791582 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.791562    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-8q5gx"]
Sep 17 18:30:04.872476 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.872455    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a40c5a6-64da-46a3-9028-c59a3a7ee147-signing-cabundle\") pod \"service-ca-55f6fc7b86-8q5gx\" (UID: \"1a40c5a6-64da-46a3-9028-c59a3a7ee147\") " pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:04.872581 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.872495    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1a40c5a6-64da-46a3-9028-c59a3a7ee147-signing-key\") pod \"service-ca-55f6fc7b86-8q5gx\" (UID: \"1a40c5a6-64da-46a3-9028-c59a3a7ee147\") " pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:04.872581 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.872524    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssdz\" (UniqueName: \"kubernetes.io/projected/1a40c5a6-64da-46a3-9028-c59a3a7ee147-kube-api-access-tssdz\") pod \"service-ca-55f6fc7b86-8q5gx\" (UID: \"1a40c5a6-64da-46a3-9028-c59a3a7ee147\") " pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:04.973128 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.973091    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a40c5a6-64da-46a3-9028-c59a3a7ee147-signing-cabundle\") pod \"service-ca-55f6fc7b86-8q5gx\" (UID: \"1a40c5a6-64da-46a3-9028-c59a3a7ee147\") " pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:04.973237 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.973154    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1a40c5a6-64da-46a3-9028-c59a3a7ee147-signing-key\") pod \"service-ca-55f6fc7b86-8q5gx\" (UID: \"1a40c5a6-64da-46a3-9028-c59a3a7ee147\") " pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:04.973237 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.973205    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssdz\" (UniqueName: \"kubernetes.io/projected/1a40c5a6-64da-46a3-9028-c59a3a7ee147-kube-api-access-tssdz\") pod \"service-ca-55f6fc7b86-8q5gx\" (UID: \"1a40c5a6-64da-46a3-9028-c59a3a7ee147\") " pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:04.973852 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.973834    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a40c5a6-64da-46a3-9028-c59a3a7ee147-signing-cabundle\") pod \"service-ca-55f6fc7b86-8q5gx\" (UID: \"1a40c5a6-64da-46a3-9028-c59a3a7ee147\") " pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:04.975532 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.975514    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1a40c5a6-64da-46a3-9028-c59a3a7ee147-signing-key\") pod \"service-ca-55f6fc7b86-8q5gx\" (UID: \"1a40c5a6-64da-46a3-9028-c59a3a7ee147\") " pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:04.981562 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:04.981543    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssdz\" (UniqueName: \"kubernetes.io/projected/1a40c5a6-64da-46a3-9028-c59a3a7ee147-kube-api-access-tssdz\") pod \"service-ca-55f6fc7b86-8q5gx\" (UID: \"1a40c5a6-64da-46a3-9028-c59a3a7ee147\") " pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:05.092972 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:05.092933    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx"
Sep 17 18:30:05.335352 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:05.335328    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-px26v_ba444217-b1b1-4f63-9579-d097b752acaa/dns-node-resolver/0.log"
Sep 17 18:30:05.364380 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:05.364352    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-8q5gx"]
Sep 17 18:30:05.366906 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:05.366877    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a40c5a6_64da_46a3_9028_c59a3a7ee147.slice/crio-fb5e1d8b6865f989a5789163e999d741ef3fbf6fef73a5fbc17e5d1585c34df6 WatchSource:0}: Error finding container fb5e1d8b6865f989a5789163e999d741ef3fbf6fef73a5fbc17e5d1585c34df6: Status 404 returned error can't find the container with id fb5e1d8b6865f989a5789163e999d741ef3fbf6fef73a5fbc17e5d1585c34df6
Sep 17 18:30:06.134644 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:06.134610    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mgf76_9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7/node-ca/0.log"
Sep 17 18:30:06.260359 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:06.260327    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx" event={"ID":"1a40c5a6-64da-46a3-9028-c59a3a7ee147","Type":"ContainerStarted","Data":"fb5e1d8b6865f989a5789163e999d741ef3fbf6fef73a5fbc17e5d1585c34df6"}
Sep 17 18:30:06.585099 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:06.585074    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:06.585550 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:06.585265    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:06.585550 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:06.585329    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:06.585550 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:06.585361    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls podName:17c175cf-5b8a-4201-bd88-88f28de96ec6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.585340878 +0000 UTC m=+51.795239031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tjv2" (UID: "17c175cf-5b8a-4201-bd88-88f28de96ec6") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:06.585550 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:06.585269    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:06.585550 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:06.585386    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm podName:17c175cf-5b8a-4201-bd88-88f28de96ec6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.585374255 +0000 UTC m=+51.795272409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4tjv2" (UID: "17c175cf-5b8a-4201-bd88-88f28de96ec6") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:06.737440 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:06.737415    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lfc2n_beecf21c-b666-4309-b7af-0165216fff83/serve-healthcheck-canary/0.log"
Sep 17 18:30:07.142906 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:07.142878    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-pppzq_260fbd36-e31c-48d5-bf5a-71f7b1d495a3/migrator/0.log"
Sep 17 18:30:07.264654 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:07.264618    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx" event={"ID":"1a40c5a6-64da-46a3-9028-c59a3a7ee147","Type":"ContainerStarted","Data":"a52296089ef2f967306bb35ee4b7a926a2075e2626c6d202367021d32553807d"}
Sep 17 18:30:07.296774 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:07.296726    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-8q5gx" podStartSLOduration=1.605623001 podStartE2EDuration="3.29671284s" totalImagesPullingTime="1.691089839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:07.294961007 +0000 UTC m=+48.504859158" watchObservedRunningTime="2026-09-17 18:30:07.29671284 +0000 UTC m=+48.506610992"
Sep 17 18:30:07.352344 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:07.352319    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-pppzq_260fbd36-e31c-48d5-bf5a-71f7b1d495a3/graceful-termination/0.log"
Sep 17 18:30:10.015848 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:10.015801    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:30:10.016232 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:10.015956    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:10.016232 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:10.016019    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls podName:a8f199ee-cba0-43b7-867f-8503480c9ff8 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:26.016002609 +0000 UTC m=+67.225900748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls") pod "dns-default-rgs4g" (UID: "a8f199ee-cba0-43b7-867f-8503480c9ff8") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:10.117068 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:10.117038    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:30:10.117201 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:10.117186    2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:10.117231 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:10.117203    2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b697ddd8-559wg: secret "image-registry-tls" not found
Sep 17 18:30:10.117273 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:10.117257    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls podName:9d4ab780-ba57-4714-b6a1-5a75369cb415 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:26.117240951 +0000 UTC m=+67.327139088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls") pod "image-registry-6b697ddd8-559wg" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415") : secret "image-registry-tls" not found
Sep 17 18:30:10.620079 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:10.620047    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:10.620248 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:10.620094    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:10.620286 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:10.620253    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:10.620352 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:30:10.620340    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls podName:17c175cf-5b8a-4201-bd88-88f28de96ec6 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:18.620320309 +0000 UTC m=+59.830218458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tjv2" (UID: "17c175cf-5b8a-4201-bd88-88f28de96ec6") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:10.620686 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:10.620665    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17c175cf-5b8a-4201-bd88-88f28de96ec6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:18.683513 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:18.683474    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:18.685723 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:18.685700    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17c175cf-5b8a-4201-bd88-88f28de96ec6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tjv2\" (UID: \"17c175cf-5b8a-4201-bd88-88f28de96ec6\") " pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:18.736361 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:18.736331    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4tjv2"
Sep 17 18:30:19.027562 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:19.027392    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4tjv2"]
Sep 17 18:30:19.030065 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:19.030039    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c175cf_5b8a_4201_bd88_88f28de96ec6.slice/crio-de3f49c3daf03350eca78bfdf60c339ccfa6f0be41627462c48eb9c77bb54adf WatchSource:0}: Error finding container de3f49c3daf03350eca78bfdf60c339ccfa6f0be41627462c48eb9c77bb54adf: Status 404 returned error can't find the container with id de3f49c3daf03350eca78bfdf60c339ccfa6f0be41627462c48eb9c77bb54adf
Sep 17 18:30:19.298593 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:19.298511    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tjv2" event={"ID":"17c175cf-5b8a-4201-bd88-88f28de96ec6","Type":"ContainerStarted","Data":"1f33d9932cb8fa13cad8863ace22cea24057abdc43dce865595650c085ae821e"}
Sep 17 18:30:19.298593 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:19.298568    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tjv2" event={"ID":"17c175cf-5b8a-4201-bd88-88f28de96ec6","Type":"ContainerStarted","Data":"de3f49c3daf03350eca78bfdf60c339ccfa6f0be41627462c48eb9c77bb54adf"}
Sep 17 18:30:20.210455 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:20.210428    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tg7sn"
Sep 17 18:30:21.307730 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:21.307696    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tjv2" event={"ID":"17c175cf-5b8a-4201-bd88-88f28de96ec6","Type":"ContainerStarted","Data":"0d7edbbd1d601f6700d5788f57102aa3e2c469ea5fc65de17dec99cf5632164b"}
Sep 17 18:30:22.312424 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:22.312333    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tjv2" event={"ID":"17c175cf-5b8a-4201-bd88-88f28de96ec6","Type":"ContainerStarted","Data":"6912e4a46cda6a79b290935c1a52b4ddbfa9775a07588952987cfe6630692658"}
Sep 17 18:30:22.339856 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:22.339809    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4tjv2" podStartSLOduration=17.695300352 podStartE2EDuration="20.339793588s" totalImagesPullingTime="2.644493236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:22.339206029 +0000 UTC m=+63.549104184" watchObservedRunningTime="2026-09-17 18:30:22.339793588 +0000 UTC m=+63.549691739"
Sep 17 18:30:26.043628 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.043595    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:30:26.045892 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.045873    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f199ee-cba0-43b7-867f-8503480c9ff8-metrics-tls\") pod \"dns-default-rgs4g\" (UID: \"a8f199ee-cba0-43b7-867f-8503480c9ff8\") " pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:30:26.144869 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.144825    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:30:26.147042 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.147026    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"image-registry-6b697ddd8-559wg\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") " pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:30:26.321087 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.321010    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cpkjn\""
Sep 17 18:30:26.329298 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.329279    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:30:26.361803 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.361774    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-26p5d\""
Sep 17 18:30:26.369220 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.369199    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:30:26.598708 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.598684    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rgs4g"]
Sep 17 18:30:26.600535 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:26.600510    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f199ee_cba0_43b7_867f_8503480c9ff8.slice/crio-ede41ae0029e13cd3c7e8ef8aec64a6dfce52158897caab7da4154c50cebf7f3 WatchSource:0}: Error finding container ede41ae0029e13cd3c7e8ef8aec64a6dfce52158897caab7da4154c50cebf7f3: Status 404 returned error can't find the container with id ede41ae0029e13cd3c7e8ef8aec64a6dfce52158897caab7da4154c50cebf7f3
Sep 17 18:30:26.648815 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.648794    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b697ddd8-559wg"]
Sep 17 18:30:26.650766 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:26.650741    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4ab780_ba57_4714_b6a1_5a75369cb415.slice/crio-144660aaf18fb0a0f45760a3f63ffd2c8792033eb5d81326fab45ffa1721cf6f WatchSource:0}: Error finding container 144660aaf18fb0a0f45760a3f63ffd2c8792033eb5d81326fab45ffa1721cf6f: Status 404 returned error can't find the container with id 144660aaf18fb0a0f45760a3f63ffd2c8792033eb5d81326fab45ffa1721cf6f
Sep 17 18:30:26.748525 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.748496    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:30:26.748684 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.748537    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:30:26.753108 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.753090    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:26.753168 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.753108    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:26.761354 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.761333    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/876396f2-26fd-4a04-a6f0-00a1e8d41c3c-metrics-certs\") pod \"network-metrics-daemon-bw2tp\" (UID: \"876396f2-26fd-4a04-a6f0-00a1e8d41c3c\") " pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:30:26.761455 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.761397    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e9337787-7c1b-4e8c-add0-59cd333bbd43-original-pull-secret\") pod \"global-pull-secret-syncer-p45zn\" (UID: \"e9337787-7c1b-4e8c-add0-59cd333bbd43\") " pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:30:26.849438 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.849345    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:30:26.853911 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.853893    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:30:26.862632 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.862617    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:30:26.872533 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.872501    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p586n\" (UniqueName: \"kubernetes.io/projected/6970b20c-83ca-4006-b799-7157e09bf405-kube-api-access-p586n\") pod \"network-check-target-g8m7p\" (UID: \"6970b20c-83ca-4006-b799-7157e09bf405\") " pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:30:26.978078 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.978044    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dzhw6\""
Sep 17 18:30:26.979089 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.979072    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p45zn"
Sep 17 18:30:26.983541 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.983360    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2tp"
Sep 17 18:30:26.986680 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.986665    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rcjbl\""
Sep 17 18:30:26.994618 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:26.994589    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:30:27.181896 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.181815    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b697ddd8-559wg"]
Sep 17 18:30:27.182344 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.181903    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"]
Sep 17 18:30:27.185717 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.185690    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"]
Sep 17 18:30:27.185857 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.185808    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.187698 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.187672    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"
Sep 17 18:30:27.189747 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.189656    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 18:30:27.189747 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.189685    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:30:27.191428 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.191408    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"]
Sep 17 18:30:27.191661 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.191638    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kxsj5\""
Sep 17 18:30:27.191755 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.191541    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:30:27.191795 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.191772    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 18:30:27.191853 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.191541    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:30:27.193979 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.193962    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.197463 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.197441    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 18:30:27.197539 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.197492    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 18:30:27.197809 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.197792    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 18:30:27.197875 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.197857    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 18:30:27.201526 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.201429    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"]
Sep 17 18:30:27.210744 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.210724    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"]
Sep 17 18:30:27.215987 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.215969    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"]
Sep 17 18:30:27.252618 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.252815 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252634    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.252815 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252674    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d1bcddf1-c3c2-4d89-9276-75a66db8da30-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bdbc85fdb-b7g4j\" (UID: \"d1bcddf1-c3c2-4d89-9276-75a66db8da30\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.252815 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-ca\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.252815 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252772    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dwk\" (UniqueName: \"kubernetes.io/projected/269d4ccb-7218-4f6c-b71c-a69721e8ec19-kube-api-access-22dwk\") pod \"managed-serviceaccount-addon-agent-59f585d76-bqfs8\" (UID: \"269d4ccb-7218-4f6c-b71c-a69721e8ec19\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"
Sep 17 18:30:27.253057 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252818    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktln\" (UniqueName: \"kubernetes.io/projected/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-kube-api-access-kktln\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.253057 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252843    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1bcddf1-c3c2-4d89-9276-75a66db8da30-tmp\") pod \"klusterlet-addon-workmgr-7bdbc85fdb-b7g4j\" (UID: \"d1bcddf1-c3c2-4d89-9276-75a66db8da30\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.253057 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252901    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-hub\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.253057 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252943    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/269d4ccb-7218-4f6c-b71c-a69721e8ec19-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59f585d76-bqfs8\" (UID: \"269d4ccb-7218-4f6c-b71c-a69721e8ec19\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"
Sep 17 18:30:27.253057 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.252971    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8smv\" (UniqueName: \"kubernetes.io/projected/d1bcddf1-c3c2-4d89-9276-75a66db8da30-kube-api-access-g8smv\") pod \"klusterlet-addon-workmgr-7bdbc85fdb-b7g4j\" (UID: \"d1bcddf1-c3c2-4d89-9276-75a66db8da30\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.253057 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.253036    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.305273 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.305248    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p45zn"]
Sep 17 18:30:27.306656 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.306630    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bw2tp"]
Sep 17 18:30:27.307167 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:27.307137    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876396f2_26fd_4a04_a6f0_00a1e8d41c3c.slice/crio-4551c19f319878b82a17ab15b654bed6d5acce421270f07fca6c604e73aa7cf3 WatchSource:0}: Error finding container 4551c19f319878b82a17ab15b654bed6d5acce421270f07fca6c604e73aa7cf3: Status 404 returned error can't find the container with id 4551c19f319878b82a17ab15b654bed6d5acce421270f07fca6c604e73aa7cf3
Sep 17 18:30:27.307480 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:27.307459    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9337787_7c1b_4e8c_add0_59cd333bbd43.slice/crio-76661f7905847eb048c0845faadb715dcac79859d8f880c4e2a53245c28f595d WatchSource:0}: Error finding container 76661f7905847eb048c0845faadb715dcac79859d8f880c4e2a53245c28f595d: Status 404 returned error can't find the container with id 76661f7905847eb048c0845faadb715dcac79859d8f880c4e2a53245c28f595d
Sep 17 18:30:27.327237 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.327217    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g8m7p"]
Sep 17 18:30:27.327367 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.327266    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p45zn" event={"ID":"e9337787-7c1b-4e8c-add0-59cd333bbd43","Type":"ContainerStarted","Data":"76661f7905847eb048c0845faadb715dcac79859d8f880c4e2a53245c28f595d"}
Sep 17 18:30:27.329115 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:27.329087    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6970b20c_83ca_4006_b799_7157e09bf405.slice/crio-d1f786689b84ba9e667fce83e5a897aaccc6cc287fb56c84359e165155e0a094 WatchSource:0}: Error finding container d1f786689b84ba9e667fce83e5a897aaccc6cc287fb56c84359e165155e0a094: Status 404 returned error can't find the container with id d1f786689b84ba9e667fce83e5a897aaccc6cc287fb56c84359e165155e0a094
Sep 17 18:30:27.329588 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.329486    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b697ddd8-559wg" event={"ID":"9d4ab780-ba57-4714-b6a1-5a75369cb415","Type":"ContainerStarted","Data":"1b6bef1f953f691214312781dd4e3f637f957eb50bdc28125802193184611c63"}
Sep 17 18:30:27.329588 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.329531    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b697ddd8-559wg" event={"ID":"9d4ab780-ba57-4714-b6a1-5a75369cb415","Type":"ContainerStarted","Data":"144660aaf18fb0a0f45760a3f63ffd2c8792033eb5d81326fab45ffa1721cf6f"}
Sep 17 18:30:27.329750 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.329650    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:30:27.331555 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.331066    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bw2tp" event={"ID":"876396f2-26fd-4a04-a6f0-00a1e8d41c3c","Type":"ContainerStarted","Data":"4551c19f319878b82a17ab15b654bed6d5acce421270f07fca6c604e73aa7cf3"}
Sep 17 18:30:27.333132 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.332966    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rgs4g" event={"ID":"a8f199ee-cba0-43b7-867f-8503480c9ff8","Type":"ContainerStarted","Data":"ede41ae0029e13cd3c7e8ef8aec64a6dfce52158897caab7da4154c50cebf7f3"}
Sep 17 18:30:27.354101 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354077    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1bcddf1-c3c2-4d89-9276-75a66db8da30-tmp\") pod \"klusterlet-addon-workmgr-7bdbc85fdb-b7g4j\" (UID: \"d1bcddf1-c3c2-4d89-9276-75a66db8da30\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.354209 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354108    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-hub\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.354209 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354135    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/269d4ccb-7218-4f6c-b71c-a69721e8ec19-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59f585d76-bqfs8\" (UID: \"269d4ccb-7218-4f6c-b71c-a69721e8ec19\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"
Sep 17 18:30:27.354209 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354160    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8smv\" (UniqueName: \"kubernetes.io/projected/d1bcddf1-c3c2-4d89-9276-75a66db8da30-kube-api-access-g8smv\") pod \"klusterlet-addon-workmgr-7bdbc85fdb-b7g4j\" (UID: \"d1bcddf1-c3c2-4d89-9276-75a66db8da30\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.354388 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354291    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.354388 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354358    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.354493 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354390    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.354493 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354440    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d1bcddf1-c3c2-4d89-9276-75a66db8da30-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bdbc85fdb-b7g4j\" (UID: \"d1bcddf1-c3c2-4d89-9276-75a66db8da30\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.354493 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354480    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1bcddf1-c3c2-4d89-9276-75a66db8da30-tmp\") pod \"klusterlet-addon-workmgr-7bdbc85fdb-b7g4j\" (UID: \"d1bcddf1-c3c2-4d89-9276-75a66db8da30\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.354641 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354500    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-ca\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.354641 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354526    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dwk\" (UniqueName: \"kubernetes.io/projected/269d4ccb-7218-4f6c-b71c-a69721e8ec19-kube-api-access-22dwk\") pod \"managed-serviceaccount-addon-agent-59f585d76-bqfs8\" (UID: \"269d4ccb-7218-4f6c-b71c-a69721e8ec19\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"
Sep 17 18:30:27.354641 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.354560    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kktln\" (UniqueName: \"kubernetes.io/projected/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-kube-api-access-kktln\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.356111 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.356086    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.357536 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.357492    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-ca\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.357675 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.357653    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/269d4ccb-7218-4f6c-b71c-a69721e8ec19-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59f585d76-bqfs8\" (UID: \"269d4ccb-7218-4f6c-b71c-a69721e8ec19\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"
Sep 17 18:30:27.357948 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.357906    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.358316 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.358279    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d1bcddf1-c3c2-4d89-9276-75a66db8da30-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bdbc85fdb-b7g4j\" (UID: \"d1bcddf1-c3c2-4d89-9276-75a66db8da30\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.358723 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.358707    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.359040 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.359020    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-hub\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.384273 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.384247    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktln\" (UniqueName: \"kubernetes.io/projected/fa69d0af-1b51-470d-a9fa-91ce9f7890a1-kube-api-access-kktln\") pod \"cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk\" (UID: \"fa69d0af-1b51-470d-a9fa-91ce9f7890a1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.389874 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.389827    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b697ddd8-559wg" podStartSLOduration=45.389809728 podStartE2EDuration="45.389809728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:27.382836244 +0000 UTC m=+68.592734397" watchObservedRunningTime="2026-09-17 18:30:27.389809728 +0000 UTC m=+68.599707876"
Sep 17 18:30:27.407349 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.407324    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dwk\" (UniqueName: \"kubernetes.io/projected/269d4ccb-7218-4f6c-b71c-a69721e8ec19-kube-api-access-22dwk\") pod \"managed-serviceaccount-addon-agent-59f585d76-bqfs8\" (UID: \"269d4ccb-7218-4f6c-b71c-a69721e8ec19\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"
Sep 17 18:30:27.407856 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.407836    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8smv\" (UniqueName: \"kubernetes.io/projected/d1bcddf1-c3c2-4d89-9276-75a66db8da30-kube-api-access-g8smv\") pod \"klusterlet-addon-workmgr-7bdbc85fdb-b7g4j\" (UID: \"d1bcddf1-c3c2-4d89-9276-75a66db8da30\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.500001 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.499914    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:27.517265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.517240    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"
Sep 17 18:30:27.524646 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.524475    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"
Sep 17 18:30:27.795235 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.795159    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"]
Sep 17 18:30:27.827867 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.827483    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8"]
Sep 17 18:30:27.872839 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:27.872772    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk"]
Sep 17 18:30:28.339317 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:28.339272    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8m7p" event={"ID":"6970b20c-83ca-4006-b799-7157e09bf405","Type":"ContainerStarted","Data":"d1f786689b84ba9e667fce83e5a897aaccc6cc287fb56c84359e165155e0a094"}
Sep 17 18:30:28.363471 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:28.363430    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod269d4ccb_7218_4f6c_b71c_a69721e8ec19.slice/crio-f71b21159685ead0e9c85c25a0ea178db436099f5b370dce1701cbeef9307d30 WatchSource:0}: Error finding container f71b21159685ead0e9c85c25a0ea178db436099f5b370dce1701cbeef9307d30: Status 404 returned error can't find the container with id f71b21159685ead0e9c85c25a0ea178db436099f5b370dce1701cbeef9307d30
Sep 17 18:30:28.364555 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:28.364522    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa69d0af_1b51_470d_a9fa_91ce9f7890a1.slice/crio-e623483e8d289838fc0752e74732b79ea98bd943a3b8b6a4c1dfb70b87108d8d WatchSource:0}: Error finding container e623483e8d289838fc0752e74732b79ea98bd943a3b8b6a4c1dfb70b87108d8d: Status 404 returned error can't find the container with id e623483e8d289838fc0752e74732b79ea98bd943a3b8b6a4c1dfb70b87108d8d
Sep 17 18:30:28.366642 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:28.366597    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bcddf1_c3c2_4d89_9276_75a66db8da30.slice/crio-2c90bda4a27ef9198d052dc541295cf7f941571153718e8e77aa7d944f31d2b9 WatchSource:0}: Error finding container 2c90bda4a27ef9198d052dc541295cf7f941571153718e8e77aa7d944f31d2b9: Status 404 returned error can't find the container with id 2c90bda4a27ef9198d052dc541295cf7f941571153718e8e77aa7d944f31d2b9
Sep 17 18:30:29.351509 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:29.349485    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8" event={"ID":"269d4ccb-7218-4f6c-b71c-a69721e8ec19","Type":"ContainerStarted","Data":"f71b21159685ead0e9c85c25a0ea178db436099f5b370dce1701cbeef9307d30"}
Sep 17 18:30:29.355014 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:29.354906    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bw2tp" event={"ID":"876396f2-26fd-4a04-a6f0-00a1e8d41c3c","Type":"ContainerStarted","Data":"48d67fdf1b1007f65d43851d15d1fc3f90f289e8f5a1b5b043f616a9952c5743"}
Sep 17 18:30:29.363809 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:29.362934    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rgs4g" event={"ID":"a8f199ee-cba0-43b7-867f-8503480c9ff8","Type":"ContainerStarted","Data":"981a9e2e9325b8454263b4983ddb11efb178c3aa4e6eb958cb5464ce391f0784"}
Sep 17 18:30:29.363809 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:29.362986    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rgs4g" event={"ID":"a8f199ee-cba0-43b7-867f-8503480c9ff8","Type":"ContainerStarted","Data":"6769ae053f4079f82cdc35e9c77606d4cba7266441821317db824dac3aab82ac"}
Sep 17 18:30:29.363809 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:29.363755    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:30:29.366077 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:29.366011    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j" event={"ID":"d1bcddf1-c3c2-4d89-9276-75a66db8da30","Type":"ContainerStarted","Data":"2c90bda4a27ef9198d052dc541295cf7f941571153718e8e77aa7d944f31d2b9"}
Sep 17 18:30:29.370141 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:29.370096    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk" event={"ID":"fa69d0af-1b51-470d-a9fa-91ce9f7890a1","Type":"ContainerStarted","Data":"e623483e8d289838fc0752e74732b79ea98bd943a3b8b6a4c1dfb70b87108d8d"}
Sep 17 18:30:29.383360 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:29.382627    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rgs4g" podStartSLOduration=33.586538456 podStartE2EDuration="35.382608508s" totalImagesPullingTime="1.796070052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:29.381385515 +0000 UTC m=+70.591283670" watchObservedRunningTime="2026-09-17 18:30:29.382608508 +0000 UTC m=+70.592506663"
Sep 17 18:30:30.384520 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:30.384429    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bw2tp" event={"ID":"876396f2-26fd-4a04-a6f0-00a1e8d41c3c","Type":"ContainerStarted","Data":"9e64fffdc982d6263641f3e35d3a460eb70d0091fec0439741a61f881586c05b"}
Sep 17 18:30:38.416024 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.415988    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p45zn" event={"ID":"e9337787-7c1b-4e8c-add0-59cd333bbd43","Type":"ContainerStarted","Data":"6ea7ec1b7ac0283b2435a7491bee7a4ead603584e06edaa89aa947b88cb4a962"}
Sep 17 18:30:38.417575 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.417554    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g8m7p" event={"ID":"6970b20c-83ca-4006-b799-7157e09bf405","Type":"ContainerStarted","Data":"2b179a1c4b80b4819b20c8f9286bf648b1be0732f4de7a72dc302073c66cb3a7"}
Sep 17 18:30:38.417684 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.417633    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:30:38.419515 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.419493    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk" event={"ID":"fa69d0af-1b51-470d-a9fa-91ce9f7890a1","Type":"ContainerStarted","Data":"bab79bddd47f986d9af72e0c73fa162e3623c8774e703eafc49f8ad1592ed7b8"}
Sep 17 18:30:38.419614 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.419525    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk" event={"ID":"fa69d0af-1b51-470d-a9fa-91ce9f7890a1","Type":"ContainerStarted","Data":"648ee30da0ee732e548d21d6551d7dceffed907e1370f37aa3159e2a7a8968e7"}
Sep 17 18:30:38.419614 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.419542    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk" event={"ID":"fa69d0af-1b51-470d-a9fa-91ce9f7890a1","Type":"ContainerStarted","Data":"c7f05e32457d9a3301b0720bab2097b74ef050a032852028d7cb83fe7b3a7e71"}
Sep 17 18:30:38.421011 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.420991    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8" event={"ID":"269d4ccb-7218-4f6c-b71c-a69721e8ec19","Type":"ContainerStarted","Data":"885bf2244863e54ec5a6f175ec122258e0801469135da2949fb3d99e158b96d5"}
Sep 17 18:30:38.422402 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.422385    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j" event={"ID":"d1bcddf1-c3c2-4d89-9276-75a66db8da30","Type":"ContainerStarted","Data":"84b07cbde0e165426c902679f1ca8cf9bff8dad410c079b5c6af66d0c34c53a7"}
Sep 17 18:30:38.422593 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.422577    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:38.424231 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.424214    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j"
Sep 17 18:30:38.433106 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.433068    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p45zn" podStartSLOduration=67.141572412 podStartE2EDuration="1m17.433057161s" totalImagesPullingTime="10.291484749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.432326388 +0000 UTC m=+79.642224538" watchObservedRunningTime="2026-09-17 18:30:38.433057161 +0000 UTC m=+79.642955313"
Sep 17 18:30:38.433243 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.433226    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bw2tp" podStartSLOduration=75.61900678 podStartE2EDuration="1m17.433220178s" totalImagesPullingTime="1.814213398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:30.430463552 +0000 UTC m=+71.640361708" watchObservedRunningTime="2026-09-17 18:30:38.433220178 +0000 UTC m=+79.643118383"
Sep 17 18:30:38.452292 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.452251    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c67b9bfdf-p2kjk" podStartSLOduration=2.208770736 podStartE2EDuration="11.452237326s" totalImagesPullingTime="9.24346659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.452000926 +0000 UTC m=+79.661899078" watchObservedRunningTime="2026-09-17 18:30:38.452237326 +0000 UTC m=+79.662135475"
Sep 17 18:30:38.475325 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.475261    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g8m7p" podStartSLOduration=67.218735729 podStartE2EDuration="1m17.475249356s" totalImagesPullingTime="10.256513627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.474641394 +0000 UTC m=+79.684539539" watchObservedRunningTime="2026-09-17 18:30:38.475249356 +0000 UTC m=+79.685147508"
Sep 17 18:30:38.493347 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.493291    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bdbc85fdb-b7g4j" podStartSLOduration=2.274368652 podStartE2EDuration="11.493278704s" totalImagesPullingTime="9.218910052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.492261573 +0000 UTC m=+79.702159727" watchObservedRunningTime="2026-09-17 18:30:38.493278704 +0000 UTC m=+79.703176859"
Sep 17 18:30:38.513669 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:38.513628    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59f585d76-bqfs8" podStartSLOduration=2.281988815 podStartE2EDuration="11.51361547s" totalImagesPullingTime="9.231626655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.512101197 +0000 UTC m=+79.721999348" watchObservedRunningTime="2026-09-17 18:30:38.51361547 +0000 UTC m=+79.723513622"
Sep 17 18:30:40.390988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.390960    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rgs4g"
Sep 17 18:30:40.937667 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.937629    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z4z75"]
Sep 17 18:30:40.940011 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.939991    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:40.943900 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.943878    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:30:40.946466 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.946324    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:30:40.946466 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.946451    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:30:40.946843 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.946826    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:30:40.948372 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.948355    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:30:40.949343 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.949327    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-24224\""
Sep 17 18:30:40.951185 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:40.951168    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:30:41.066148 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.066119    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/561e4771-4acc-42d1-86ae-51ab55a1384e-sys\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.066148 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.066153    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-wtmp\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.066333 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.066171    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g59px\" (UniqueName: \"kubernetes.io/projected/561e4771-4acc-42d1-86ae-51ab55a1384e-kube-api-access-g59px\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.066333 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.066186    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/561e4771-4acc-42d1-86ae-51ab55a1384e-metrics-client-ca\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.066333 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.066251    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-textfile\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.066333 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.066276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.066470 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.066334    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/561e4771-4acc-42d1-86ae-51ab55a1384e-root\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.066470 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.066366    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.066470 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.066392    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-tls\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.166635 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166608    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.166794 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166640    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-tls\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.166794 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166661    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/561e4771-4acc-42d1-86ae-51ab55a1384e-sys\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.166794 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166711    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/561e4771-4acc-42d1-86ae-51ab55a1384e-sys\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.166794 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166734    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-wtmp\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.166794 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166749    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g59px\" (UniqueName: \"kubernetes.io/projected/561e4771-4acc-42d1-86ae-51ab55a1384e-kube-api-access-g59px\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.166794 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166763    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/561e4771-4acc-42d1-86ae-51ab55a1384e-metrics-client-ca\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.166794 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166787    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-textfile\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.167162 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166804    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.167162 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166876    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-wtmp\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.167162 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.166945    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/561e4771-4acc-42d1-86ae-51ab55a1384e-root\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.167162 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.167017    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/561e4771-4acc-42d1-86ae-51ab55a1384e-root\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.167371 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.167238    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-textfile\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.167371 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.167274    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.167511 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.167485    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/561e4771-4acc-42d1-86ae-51ab55a1384e-metrics-client-ca\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.169261 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.169238    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.169376 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.169290    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/561e4771-4acc-42d1-86ae-51ab55a1384e-node-exporter-tls\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.181843 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.181820    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g59px\" (UniqueName: \"kubernetes.io/projected/561e4771-4acc-42d1-86ae-51ab55a1384e-kube-api-access-g59px\") pod \"node-exporter-z4z75\" (UID: \"561e4771-4acc-42d1-86ae-51ab55a1384e\") " pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.248951 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.248903    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4z75"
Sep 17 18:30:41.258294 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:30:41.258275    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561e4771_4acc_42d1_86ae_51ab55a1384e.slice/crio-e7c3dc45f8cea261cabeee02b393ee07d3f7fb48b26f4017c66120578ec30431 WatchSource:0}: Error finding container e7c3dc45f8cea261cabeee02b393ee07d3f7fb48b26f4017c66120578ec30431: Status 404 returned error can't find the container with id e7c3dc45f8cea261cabeee02b393ee07d3f7fb48b26f4017c66120578ec30431
Sep 17 18:30:41.434802 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:41.434769    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4z75" event={"ID":"561e4771-4acc-42d1-86ae-51ab55a1384e","Type":"ContainerStarted","Data":"e7c3dc45f8cea261cabeee02b393ee07d3f7fb48b26f4017c66120578ec30431"}
Sep 17 18:30:42.439793 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:42.439766    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4z75" event={"ID":"561e4771-4acc-42d1-86ae-51ab55a1384e","Type":"ContainerStarted","Data":"3367f775601058d40b0eb30af971c1655c4056e579a34e75c451e04ab1509b88"}
Sep 17 18:30:43.444375 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:43.444342    2552 generic.go:356] "Generic (PLEG): container finished" podID="561e4771-4acc-42d1-86ae-51ab55a1384e" containerID="3367f775601058d40b0eb30af971c1655c4056e579a34e75c451e04ab1509b88" exitCode=0
Sep 17 18:30:43.444375 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:43.444378    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4z75" event={"ID":"561e4771-4acc-42d1-86ae-51ab55a1384e","Type":"ContainerDied","Data":"3367f775601058d40b0eb30af971c1655c4056e579a34e75c451e04ab1509b88"}
Sep 17 18:30:44.450809 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:44.450769    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4z75" event={"ID":"561e4771-4acc-42d1-86ae-51ab55a1384e","Type":"ContainerStarted","Data":"9579354712c369db9a0293d91f32934a1e4d702a48a3e6fd237dbb1f6a6a01d9"}
Sep 17 18:30:44.451272 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:44.450866    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4z75" event={"ID":"561e4771-4acc-42d1-86ae-51ab55a1384e","Type":"ContainerStarted","Data":"76ce28e32b19257c15a13f7d3d46ebb3bcc7533ff63cb6e2084e442b44e7f0a3"}
Sep 17 18:30:44.471408 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:44.471346    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z4z75" podStartSLOduration=3.304186119 podStartE2EDuration="4.47132631s" totalImagesPullingTime="1.103948449s" totalInitContainerRuntime="63.191742ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:44.469921268 +0000 UTC m=+85.679819420" watchObservedRunningTime="2026-09-17 18:30:44.47132631 +0000 UTC m=+85.681224459"
Sep 17 18:30:48.345757 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:48.345731    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:30:53.365620 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.365580    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b697ddd8-559wg" podUID="9d4ab780-ba57-4714-b6a1-5a75369cb415" containerName="registry" containerID="cri-o://1b6bef1f953f691214312781dd4e3f637f957eb50bdc28125802193184611c63" gracePeriod=30
Sep 17 18:30:53.483131 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.483093    2552 generic.go:356] "Generic (PLEG): container finished" podID="9d4ab780-ba57-4714-b6a1-5a75369cb415" containerID="1b6bef1f953f691214312781dd4e3f637f957eb50bdc28125802193184611c63" exitCode=0
Sep 17 18:30:53.483251 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.483161    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b697ddd8-559wg" event={"ID":"9d4ab780-ba57-4714-b6a1-5a75369cb415","Type":"ContainerDied","Data":"1b6bef1f953f691214312781dd4e3f637f957eb50bdc28125802193184611c63"}
Sep 17 18:30:53.604138 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.604116    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:30:53.757718 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.757643    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-image-registry-private-configuration\") pod \"9d4ab780-ba57-4714-b6a1-5a75369cb415\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") "
Sep 17 18:30:53.757718 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.757683    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-certificates\") pod \"9d4ab780-ba57-4714-b6a1-5a75369cb415\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") "
Sep 17 18:30:53.757718 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.757707    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4ab780-ba57-4714-b6a1-5a75369cb415-ca-trust-extracted\") pod \"9d4ab780-ba57-4714-b6a1-5a75369cb415\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") "
Sep 17 18:30:53.757990 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.757727    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-installation-pull-secrets\") pod \"9d4ab780-ba57-4714-b6a1-5a75369cb415\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") "
Sep 17 18:30:53.757990 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.757744    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s9qx\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-kube-api-access-5s9qx\") pod \"9d4ab780-ba57-4714-b6a1-5a75369cb415\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") "
Sep 17 18:30:53.757990 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.757800    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-trusted-ca\") pod \"9d4ab780-ba57-4714-b6a1-5a75369cb415\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") "
Sep 17 18:30:53.757990 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.757827    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") pod \"9d4ab780-ba57-4714-b6a1-5a75369cb415\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") "
Sep 17 18:30:53.757990 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.757851    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-bound-sa-token\") pod \"9d4ab780-ba57-4714-b6a1-5a75369cb415\" (UID: \"9d4ab780-ba57-4714-b6a1-5a75369cb415\") "
Sep 17 18:30:53.758667 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.758263    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9d4ab780-ba57-4714-b6a1-5a75369cb415" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:53.758667 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.758630    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9d4ab780-ba57-4714-b6a1-5a75369cb415" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:53.760349 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.760320    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9d4ab780-ba57-4714-b6a1-5a75369cb415" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:53.760467 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.760354    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9d4ab780-ba57-4714-b6a1-5a75369cb415" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:53.760467 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.760427    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9d4ab780-ba57-4714-b6a1-5a75369cb415" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:53.760579 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.760538    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-kube-api-access-5s9qx" (OuterVolumeSpecName: "kube-api-access-5s9qx") pod "9d4ab780-ba57-4714-b6a1-5a75369cb415" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415"). InnerVolumeSpecName "kube-api-access-5s9qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:53.760633 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.760603    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9d4ab780-ba57-4714-b6a1-5a75369cb415" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:53.766659 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.766635    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4ab780-ba57-4714-b6a1-5a75369cb415-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9d4ab780-ba57-4714-b6a1-5a75369cb415" (UID: "9d4ab780-ba57-4714-b6a1-5a75369cb415"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:53.858422 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.858401    2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-trusted-ca\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:30:53.858422 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.858420    2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:30:53.858544 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.858430    2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-bound-sa-token\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:30:53.858544 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.858440    2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-image-registry-private-configuration\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:30:53.858544 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.858449    2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d4ab780-ba57-4714-b6a1-5a75369cb415-registry-certificates\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:30:53.858544 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.858458    2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d4ab780-ba57-4714-b6a1-5a75369cb415-ca-trust-extracted\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:30:53.858544 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.858466    2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d4ab780-ba57-4714-b6a1-5a75369cb415-installation-pull-secrets\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:30:53.858544 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:53.858474    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s9qx\" (UniqueName: \"kubernetes.io/projected/9d4ab780-ba57-4714-b6a1-5a75369cb415-kube-api-access-5s9qx\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:30:54.487999 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:54.487971    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b697ddd8-559wg"
Sep 17 18:30:54.487999 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:54.487982    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b697ddd8-559wg" event={"ID":"9d4ab780-ba57-4714-b6a1-5a75369cb415","Type":"ContainerDied","Data":"144660aaf18fb0a0f45760a3f63ffd2c8792033eb5d81326fab45ffa1721cf6f"}
Sep 17 18:30:54.488531 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:54.488045    2552 scope.go:117] "RemoveContainer" containerID="1b6bef1f953f691214312781dd4e3f637f957eb50bdc28125802193184611c63"
Sep 17 18:30:54.511957 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:54.511932    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b697ddd8-559wg"]
Sep 17 18:30:54.515854 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:54.515833    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b697ddd8-559wg"]
Sep 17 18:30:55.062739 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:30:55.062703    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4ab780-ba57-4714-b6a1-5a75369cb415" path="/var/lib/kubelet/pods/9d4ab780-ba57-4714-b6a1-5a75369cb415/volumes"
Sep 17 18:31:09.428282 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:31:09.428249    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g8m7p"
Sep 17 18:34:20.942065 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:34:20.942036    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:34:20.947989 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:34:20.947968    2552 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:34:20.951993 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:34:20.951974    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:37:44.489498 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.489464    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-dwbl9"]
Sep 17 18:37:44.490064 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.489780    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4ab780-ba57-4714-b6a1-5a75369cb415" containerName="registry"
Sep 17 18:37:44.490064 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.489795    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4ab780-ba57-4714-b6a1-5a75369cb415" containerName="registry"
Sep 17 18:37:44.490064 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.489847    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4ab780-ba57-4714-b6a1-5a75369cb415" containerName="registry"
Sep 17 18:37:44.491512 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.491496    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.494694 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.494673    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:37:44.494830 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.494803    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 18:37:44.494830 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.494816    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 18:37:44.494936 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.494842    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:37:44.495948 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.495933    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-t7vfh\""
Sep 17 18:37:44.508953 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.508929    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-dwbl9"]
Sep 17 18:37:44.605073 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.605047    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e61e9db-f164-4a2d-a90a-81f92f15a015-cert\") pod \"odh-model-controller-7b966d7984-dwbl9\" (UID: \"1e61e9db-f164-4a2d-a90a-81f92f15a015\") " pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.605196 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.605097    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr2h2\" (UniqueName: \"kubernetes.io/projected/1e61e9db-f164-4a2d-a90a-81f92f15a015-kube-api-access-cr2h2\") pod \"odh-model-controller-7b966d7984-dwbl9\" (UID: \"1e61e9db-f164-4a2d-a90a-81f92f15a015\") " pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.605196 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.605127    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e61e9db-f164-4a2d-a90a-81f92f15a015-metrics-certs\") pod \"odh-model-controller-7b966d7984-dwbl9\" (UID: \"1e61e9db-f164-4a2d-a90a-81f92f15a015\") " pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.706418 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.706394    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2h2\" (UniqueName: \"kubernetes.io/projected/1e61e9db-f164-4a2d-a90a-81f92f15a015-kube-api-access-cr2h2\") pod \"odh-model-controller-7b966d7984-dwbl9\" (UID: \"1e61e9db-f164-4a2d-a90a-81f92f15a015\") " pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.706542 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.706426    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e61e9db-f164-4a2d-a90a-81f92f15a015-metrics-certs\") pod \"odh-model-controller-7b966d7984-dwbl9\" (UID: \"1e61e9db-f164-4a2d-a90a-81f92f15a015\") " pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.706542 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.706467    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e61e9db-f164-4a2d-a90a-81f92f15a015-cert\") pod \"odh-model-controller-7b966d7984-dwbl9\" (UID: \"1e61e9db-f164-4a2d-a90a-81f92f15a015\") " pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.708713 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.708696    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e61e9db-f164-4a2d-a90a-81f92f15a015-metrics-certs\") pod \"odh-model-controller-7b966d7984-dwbl9\" (UID: \"1e61e9db-f164-4a2d-a90a-81f92f15a015\") " pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.708808 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.708792    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e61e9db-f164-4a2d-a90a-81f92f15a015-cert\") pod \"odh-model-controller-7b966d7984-dwbl9\" (UID: \"1e61e9db-f164-4a2d-a90a-81f92f15a015\") " pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.716385 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.716360    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2h2\" (UniqueName: \"kubernetes.io/projected/1e61e9db-f164-4a2d-a90a-81f92f15a015-kube-api-access-cr2h2\") pod \"odh-model-controller-7b966d7984-dwbl9\" (UID: \"1e61e9db-f164-4a2d-a90a-81f92f15a015\") " pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:44.802040 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:44.801990    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:45.069979 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:45.069905    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-dwbl9"]
Sep 17 18:37:45.072373 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:37:45.072348    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e61e9db_f164_4a2d_a90a_81f92f15a015.slice/crio-a7826eaa03450cd28c443f0aa552ad73553fc19b64744d0d180bfee451e6e558 WatchSource:0}: Error finding container a7826eaa03450cd28c443f0aa552ad73553fc19b64744d0d180bfee451e6e558: Status 404 returned error can't find the container with id a7826eaa03450cd28c443f0aa552ad73553fc19b64744d0d180bfee451e6e558
Sep 17 18:37:45.073469 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:45.073454    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:37:45.838883 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:45.838851    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-dwbl9" event={"ID":"1e61e9db-f164-4a2d-a90a-81f92f15a015","Type":"ContainerStarted","Data":"a7826eaa03450cd28c443f0aa552ad73553fc19b64744d0d180bfee451e6e558"}
Sep 17 18:37:48.852356 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:48.852328    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-dwbl9" event={"ID":"1e61e9db-f164-4a2d-a90a-81f92f15a015","Type":"ContainerStarted","Data":"7e9a3541aee7f28f7dc0fec4705f4fb6b578f8076e2133bae9f5cc2f91a61964"}
Sep 17 18:37:48.852718 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:48.852441    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:37:48.870670 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:48.870626    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-dwbl9" podStartSLOduration=2.03015581 podStartE2EDuration="4.870610158s" totalImagesPullingTime="2.840454348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:48.868676052 +0000 UTC m=+510.078574216" watchObservedRunningTime="2026-09-17 18:37:48.870610158 +0000 UTC m=+510.080508310"
Sep 17 18:37:59.857731 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:37:59.857699    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-dwbl9"
Sep 17 18:38:00.721084 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.721038    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-q7tsf"]
Sep 17 18:38:00.724721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.724705    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-q7tsf"
Sep 17 18:38:00.727437 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.727414    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:38:00.727437 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.727434    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-87248\""
Sep 17 18:38:00.727625 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.727434    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:38:00.733054 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.733035    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-q7tsf"]
Sep 17 18:38:00.818088 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.818060    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-model-cache\") pod \"s3-init-q7tsf\" (UID: \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\") " pod="kserve/s3-init-q7tsf"
Sep 17 18:38:00.818226 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.818111    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mf2g\" (UniqueName: \"kubernetes.io/projected/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-kube-api-access-8mf2g\") pod \"s3-init-q7tsf\" (UID: \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\") " pod="kserve/s3-init-q7tsf"
Sep 17 18:38:00.918677 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.918651    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-model-cache\") pod \"s3-init-q7tsf\" (UID: \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\") " pod="kserve/s3-init-q7tsf"
Sep 17 18:38:00.919036 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.918708    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mf2g\" (UniqueName: \"kubernetes.io/projected/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-kube-api-access-8mf2g\") pod \"s3-init-q7tsf\" (UID: \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\") " pod="kserve/s3-init-q7tsf"
Sep 17 18:38:00.919036 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.918993    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-model-cache\") pod \"s3-init-q7tsf\" (UID: \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\") " pod="kserve/s3-init-q7tsf"
Sep 17 18:38:00.931873 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:00.931845    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mf2g\" (UniqueName: \"kubernetes.io/projected/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-kube-api-access-8mf2g\") pod \"s3-init-q7tsf\" (UID: \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\") " pod="kserve/s3-init-q7tsf"
Sep 17 18:38:01.033498 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:01.033435    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-q7tsf"
Sep 17 18:38:01.300536 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:01.300466    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-q7tsf"]
Sep 17 18:38:01.303647 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:38:01.303623    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d8cc8fb_2099_4e6e_9aac_312ee458c17b.slice/crio-73322d66cdcf0fcb66db104d40c692acc8f3bdb5c188ed2a24fdd80ab7eeff0f WatchSource:0}: Error finding container 73322d66cdcf0fcb66db104d40c692acc8f3bdb5c188ed2a24fdd80ab7eeff0f: Status 404 returned error can't find the container with id 73322d66cdcf0fcb66db104d40c692acc8f3bdb5c188ed2a24fdd80ab7eeff0f
Sep 17 18:38:01.896964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:01.896930    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-q7tsf" event={"ID":"8d8cc8fb-2099-4e6e-9aac-312ee458c17b","Type":"ContainerStarted","Data":"73322d66cdcf0fcb66db104d40c692acc8f3bdb5c188ed2a24fdd80ab7eeff0f"}
Sep 17 18:38:05.914798 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:05.914762    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-q7tsf" event={"ID":"8d8cc8fb-2099-4e6e-9aac-312ee458c17b","Type":"ContainerStarted","Data":"d64b8f2a308db5a52c68412ea5e66f03c78d4c32ea440a19b1d132d28f0761c0"}
Sep 17 18:38:07.923854 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:07.923821    2552 generic.go:356] "Generic (PLEG): container finished" podID="8d8cc8fb-2099-4e6e-9aac-312ee458c17b" containerID="d64b8f2a308db5a52c68412ea5e66f03c78d4c32ea440a19b1d132d28f0761c0" exitCode=0
Sep 17 18:38:07.924242 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:07.923883    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-q7tsf" event={"ID":"8d8cc8fb-2099-4e6e-9aac-312ee458c17b","Type":"ContainerDied","Data":"d64b8f2a308db5a52c68412ea5e66f03c78d4c32ea440a19b1d132d28f0761c0"}
Sep 17 18:38:12.949695 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:12.949662    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-q7tsf" event={"ID":"8d8cc8fb-2099-4e6e-9aac-312ee458c17b","Type":"ContainerStarted","Data":"f04612ce9df6c60165463aaddd40ae1737b906bb3f03a9341d151bdc0eaafccb"}
Sep 17 18:38:12.970031 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:38:12.969989    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-q7tsf" podStartSLOduration=2.8660983570000003 podStartE2EDuration="12.969976348s" totalImagesPullingTime="8.367712235s" totalInitContainerRuntime="1.736165756s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:12.96815633 +0000 UTC m=+534.178054483" watchObservedRunningTime="2026-09-17 18:38:12.969976348 +0000 UTC m=+534.179874500"
Sep 17 18:39:18.159432 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:18.159397    2552 generic.go:356] "Generic (PLEG): container finished" podID="8d8cc8fb-2099-4e6e-9aac-312ee458c17b" containerID="f04612ce9df6c60165463aaddd40ae1737b906bb3f03a9341d151bdc0eaafccb" exitCode=0
Sep 17 18:39:18.159835 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:18.159462    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-q7tsf" event={"ID":"8d8cc8fb-2099-4e6e-9aac-312ee458c17b","Type":"ContainerDied","Data":"f04612ce9df6c60165463aaddd40ae1737b906bb3f03a9341d151bdc0eaafccb"}
Sep 17 18:39:19.278441 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:19.278414    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-q7tsf"
Sep 17 18:39:19.335111 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:19.335090    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mf2g\" (UniqueName: \"kubernetes.io/projected/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-kube-api-access-8mf2g\") pod \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\" (UID: \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\") "
Sep 17 18:39:19.335221 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:19.335126    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-model-cache\") pod \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\" (UID: \"8d8cc8fb-2099-4e6e-9aac-312ee458c17b\") "
Sep 17 18:39:19.337135 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:19.337109    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-kube-api-access-8mf2g" (OuterVolumeSpecName: "kube-api-access-8mf2g") pod "8d8cc8fb-2099-4e6e-9aac-312ee458c17b" (UID: "8d8cc8fb-2099-4e6e-9aac-312ee458c17b"). InnerVolumeSpecName "kube-api-access-8mf2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:19.357024 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:19.357001    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d8cc8fb-2099-4e6e-9aac-312ee458c17b" (UID: "8d8cc8fb-2099-4e6e-9aac-312ee458c17b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:19.436422 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:19.436369    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mf2g\" (UniqueName: \"kubernetes.io/projected/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-kube-api-access-8mf2g\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:39:19.436422 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:19.436395    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d8cc8fb-2099-4e6e-9aac-312ee458c17b-model-cache\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:39:20.167638 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:20.167603    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-q7tsf" event={"ID":"8d8cc8fb-2099-4e6e-9aac-312ee458c17b","Type":"ContainerDied","Data":"73322d66cdcf0fcb66db104d40c692acc8f3bdb5c188ed2a24fdd80ab7eeff0f"}
Sep 17 18:39:20.167638 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:20.167642    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73322d66cdcf0fcb66db104d40c692acc8f3bdb5c188ed2a24fdd80ab7eeff0f"
Sep 17 18:39:20.167831 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:20.167614    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-q7tsf"
Sep 17 18:39:20.964478 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:20.964453    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:39:20.974826 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:20.974799    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:39:32.864198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.864107    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"]
Sep 17 18:39:32.864761 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.864516    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d8cc8fb-2099-4e6e-9aac-312ee458c17b" containerName="download-hf-model"
Sep 17 18:39:32.864761 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.864534    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d8cc8fb-2099-4e6e-9aac-312ee458c17b" containerName="download-hf-model"
Sep 17 18:39:32.864761 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.864559    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d8cc8fb-2099-4e6e-9aac-312ee458c17b" containerName="s3-init"
Sep 17 18:39:32.864761 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.864567    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d8cc8fb-2099-4e6e-9aac-312ee458c17b" containerName="s3-init"
Sep 17 18:39:32.864761 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.864632    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d8cc8fb-2099-4e6e-9aac-312ee458c17b" containerName="s3-init"
Sep 17 18:39:32.867931 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.867907    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:32.872360 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.872319    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Sep 17 18:39:32.872482 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.872323    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Sep 17 18:39:32.872482 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.872327    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:39:32.873388 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.873263    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c2hvk\""
Sep 17 18:39:32.873388 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.873331    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:39:32.873682 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.873668    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:39:32.874818 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.874600    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:39:32.901460 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.901434    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"]
Sep 17 18:39:32.931086 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.931060    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:32.931189 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.931100    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmpjv\" (UniqueName: \"kubernetes.io/projected/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-api-access-xmpjv\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:32.931189 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.931134    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad47c559-8e84-43f5-a715-f5fd2acae340-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:32.931323 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.931200    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:32.931323 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.931229    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad47c559-8e84-43f5-a715-f5fd2acae340-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:32.996923 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.996897    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"]
Sep 17 18:39:32.999963 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:32.999945    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.002666 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.002648    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e0a45-kube-rbac-proxy-sar-config\""
Sep 17 18:39:33.002753 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.002648    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e0a45-predictor-serving-cert\""
Sep 17 18:39:33.032464 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.032439    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.032553 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.032471    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmpjv\" (UniqueName: \"kubernetes.io/projected/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-api-access-xmpjv\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.032553 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.032492    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad47c559-8e84-43f5-a715-f5fd2acae340-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.032553 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.032523    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.032713 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.032656    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad47c559-8e84-43f5-a715-f5fd2acae340-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.032801 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:39:33.032779    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 17 18:39:33.032869 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:39:33.032858    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad47c559-8e84-43f5-a715-f5fd2acae340-proxy-tls podName:ad47c559-8e84-43f5-a715-f5fd2acae340 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:33.532838801 +0000 UTC m=+614.742736939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad47c559-8e84-43f5-a715-f5fd2acae340-proxy-tls") pod "isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" (UID: "ad47c559-8e84-43f5-a715-f5fd2acae340") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Sep 17 18:39:33.032926 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.032880    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad47c559-8e84-43f5-a715-f5fd2acae340-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.033081 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.033065    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.033177 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.033161    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.037668 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.037644    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"]
Sep 17 18:39:33.054972 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.054951    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmpjv\" (UniqueName: \"kubernetes.io/projected/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-api-access-xmpjv\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.133863 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.133800    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") " pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.133964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.133933    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-proxy-tls\") pod \"success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") " pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.134017 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.133977    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg78b\" (UniqueName: \"kubernetes.io/projected/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-api-access-vg78b\") pod \"success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") " pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.234765 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.234736    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") " pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.234887 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.234777    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-proxy-tls\") pod \"success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") " pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.234887 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.234802    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg78b\" (UniqueName: \"kubernetes.io/projected/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-api-access-vg78b\") pod \"success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") " pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.235350 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.235330    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") " pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.237202 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.237177    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-proxy-tls\") pod \"success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") " pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.252740 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.252710    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg78b\" (UniqueName: \"kubernetes.io/projected/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-api-access-vg78b\") pod \"success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") " pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.309174 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.309145    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:39:33.401779 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.401723    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"]
Sep 17 18:39:33.405034 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.405019    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.408763 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.408745    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Sep 17 18:39:33.408844 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.408744    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Sep 17 18:39:33.422175 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.422149    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"]
Sep 17 18:39:33.537032 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.537000    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.537188 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.537041    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad47c559-8e84-43f5-a715-f5fd2acae340-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.537188 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.537067    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkrx\" (UniqueName: \"kubernetes.io/projected/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-api-access-2nkrx\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.537188 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.537108    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.537188 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.537126    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.537423 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.537183    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.539453 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.539433    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad47c559-8e84-43f5-a715-f5fd2acae340-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.593161 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.593133    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"]
Sep 17 18:39:33.596282 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:39:33.596259    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e331d9_d88e_4ede_aaaa_4cb67fb35bcb.slice/crio-4ff9ce29ce36d378436cf4193ec9dd74c270d7a45be0d5478595107147dced9f WatchSource:0}: Error finding container 4ff9ce29ce36d378436cf4193ec9dd74c270d7a45be0d5478595107147dced9f: Status 404 returned error can't find the container with id 4ff9ce29ce36d378436cf4193ec9dd74c270d7a45be0d5478595107147dced9f
Sep 17 18:39:33.637806 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.637784    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.637917 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.637814    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.637917 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.637850    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.637917 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.637886    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkrx\" (UniqueName: \"kubernetes.io/projected/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-api-access-2nkrx\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.638072 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.637947    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.638242 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.638217    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.638531 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.638515    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.638588 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.638559    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.640200 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.640182    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.658136 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.658080    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkrx\" (UniqueName: \"kubernetes.io/projected/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-api-access-2nkrx\") pod \"isvc-xgboost-graph-predictor-555fc69d56-n9npf\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.714479 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.714448    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:39:33.779106 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.779053    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:39:33.847148 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.847122    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"]
Sep 17 18:39:33.850617 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.850595    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:33.854442 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.854419    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Sep 17 18:39:33.856930 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.856844    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Sep 17 18:39:33.940595 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.940530    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a04390-4d7a-4489-bcde-f9d4161d3174-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:33.940595 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.940586    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:33.940979 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.940723    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a04390-4d7a-4489-bcde-f9d4161d3174-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:33.940979 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.940772    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:33.940979 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.940800    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbb68\" (UniqueName: \"kubernetes.io/projected/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-api-access-fbb68\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:33.989958 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:33.989935    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"]
Sep 17 18:39:34.042077 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.042050    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a04390-4d7a-4489-bcde-f9d4161d3174-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.042208 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.042085    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.042208 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.042198    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a04390-4d7a-4489-bcde-f9d4161d3174-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.042460 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.042243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.042460 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.042353    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbb68\" (UniqueName: \"kubernetes.io/projected/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-api-access-fbb68\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.042460 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:39:34.042388    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 17 18:39:34.042460 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:39:34.042452    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a04390-4d7a-4489-bcde-f9d4161d3174-proxy-tls podName:60a04390-4d7a-4489-bcde-f9d4161d3174 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:34.54243177 +0000 UTC m=+615.752329922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/60a04390-4d7a-4489-bcde-f9d4161d3174-proxy-tls") pod "isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" (UID: "60a04390-4d7a-4489-bcde-f9d4161d3174") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Sep 17 18:39:34.042664 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.042470    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a04390-4d7a-4489-bcde-f9d4161d3174-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.042850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.042830    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.042885 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.042830    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.049548 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.049529    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"]
Sep 17 18:39:34.051453 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:39:34.051434    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5d0f1b_a649_4c94_a4fe_a6d8103be06d.slice/crio-ca6a86ed1459f84be31b928f729745c2613e7e67d851f82f25405765b33b0bda WatchSource:0}: Error finding container ca6a86ed1459f84be31b928f729745c2613e7e67d851f82f25405765b33b0bda: Status 404 returned error can't find the container with id ca6a86ed1459f84be31b928f729745c2613e7e67d851f82f25405765b33b0bda
Sep 17 18:39:34.066330 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.066289    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbb68\" (UniqueName: \"kubernetes.io/projected/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-api-access-fbb68\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.089664 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.089636    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"]
Sep 17 18:39:34.094648 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:39:34.094626    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad47c559_8e84_43f5_a715_f5fd2acae340.slice/crio-ced9771637b44270234f98ecaaf4f3302cb23e689974396859f0d4911082180c WatchSource:0}: Error finding container ced9771637b44270234f98ecaaf4f3302cb23e689974396859f0d4911082180c: Status 404 returned error can't find the container with id ced9771637b44270234f98ecaaf4f3302cb23e689974396859f0d4911082180c
Sep 17 18:39:34.227546 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.227388    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" event={"ID":"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb","Type":"ContainerStarted","Data":"4ff9ce29ce36d378436cf4193ec9dd74c270d7a45be0d5478595107147dced9f"}
Sep 17 18:39:34.234252 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.234219    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" event={"ID":"ad47c559-8e84-43f5-a715-f5fd2acae340","Type":"ContainerStarted","Data":"3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df"}
Sep 17 18:39:34.234409 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.234289    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" event={"ID":"ad47c559-8e84-43f5-a715-f5fd2acae340","Type":"ContainerStarted","Data":"ced9771637b44270234f98ecaaf4f3302cb23e689974396859f0d4911082180c"}
Sep 17 18:39:34.253948 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.253922    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" event={"ID":"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d","Type":"ContainerStarted","Data":"077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2"}
Sep 17 18:39:34.254062 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.253981    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" event={"ID":"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d","Type":"ContainerStarted","Data":"ca6a86ed1459f84be31b928f729745c2613e7e67d851f82f25405765b33b0bda"}
Sep 17 18:39:34.547657 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.547055    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a04390-4d7a-4489-bcde-f9d4161d3174-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.556329 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.556213    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a04390-4d7a-4489-bcde-f9d4161d3174-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:34.762445 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:34.762382    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:39:35.132099 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:39:35.130736    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a04390_4d7a_4489_bcde_f9d4161d3174.slice/crio-5320088dd74a2b4895181524899f7333aba6f461835e50e3f42e83734eef6304 WatchSource:0}: Error finding container 5320088dd74a2b4895181524899f7333aba6f461835e50e3f42e83734eef6304: Status 404 returned error can't find the container with id 5320088dd74a2b4895181524899f7333aba6f461835e50e3f42e83734eef6304
Sep 17 18:39:35.132099 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:35.131622    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"]
Sep 17 18:39:35.268244 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:35.268105    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" event={"ID":"60a04390-4d7a-4489-bcde-f9d4161d3174","Type":"ContainerStarted","Data":"743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f"}
Sep 17 18:39:35.268244 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:35.268181    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" event={"ID":"60a04390-4d7a-4489-bcde-f9d4161d3174","Type":"ContainerStarted","Data":"5320088dd74a2b4895181524899f7333aba6f461835e50e3f42e83734eef6304"}
Sep 17 18:39:39.291397 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:39.291361    2552 generic.go:356] "Generic (PLEG): container finished" podID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerID="3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df" exitCode=0
Sep 17 18:39:39.291851 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:39.291449    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" event={"ID":"ad47c559-8e84-43f5-a715-f5fd2acae340","Type":"ContainerDied","Data":"3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df"}
Sep 17 18:39:39.293910 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:39.293879    2552 generic.go:356] "Generic (PLEG): container finished" podID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerID="077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2" exitCode=0
Sep 17 18:39:39.294030 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:39.293927    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" event={"ID":"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d","Type":"ContainerDied","Data":"077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2"}
Sep 17 18:39:40.306713 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:40.306676    2552 generic.go:356] "Generic (PLEG): container finished" podID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerID="743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f" exitCode=0
Sep 17 18:39:40.307168 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:39:40.306760    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" event={"ID":"60a04390-4d7a-4489-bcde-f9d4161d3174","Type":"ContainerDied","Data":"743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f"}
Sep 17 18:40:05.444257 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:05.444212    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" event={"ID":"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb","Type":"ContainerStarted","Data":"b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08"}
Sep 17 18:40:05.466337 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:05.456144    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" event={"ID":"60a04390-4d7a-4489-bcde-f9d4161d3174","Type":"ContainerStarted","Data":"2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9"}
Sep 17 18:40:05.466337 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:05.458798    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" event={"ID":"ad47c559-8e84-43f5-a715-f5fd2acae340","Type":"ContainerStarted","Data":"64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517"}
Sep 17 18:40:05.466337 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:05.463936    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" event={"ID":"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d","Type":"ContainerStarted","Data":"a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8"}
Sep 17 18:40:08.482021 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.481988    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" event={"ID":"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb","Type":"ContainerStarted","Data":"c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e"}
Sep 17 18:40:08.482481 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.482227    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:40:08.482481 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.482371    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:40:08.483870 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.483828    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 18:40:08.484359 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.484325    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" event={"ID":"60a04390-4d7a-4489-bcde-f9d4161d3174","Type":"ContainerStarted","Data":"2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659"}
Sep 17 18:40:08.484474 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.484442    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:40:08.486478 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.486459    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" event={"ID":"ad47c559-8e84-43f5-a715-f5fd2acae340","Type":"ContainerStarted","Data":"61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37"}
Sep 17 18:40:08.486653 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.486640    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:40:08.486721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.486680    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:40:08.487712 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.487679    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:40:08.488791 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.488774    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" event={"ID":"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d","Type":"ContainerStarted","Data":"817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6"}
Sep 17 18:40:08.488913 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.488902    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:40:08.505920 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.505878    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podStartSLOduration=10.715054203 podStartE2EDuration="36.505865614s" totalImagesPullingTime="25.790811411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:08.504674765 +0000 UTC m=+649.714572920" watchObservedRunningTime="2026-09-17 18:40:08.505865614 +0000 UTC m=+649.715763839"
Sep 17 18:40:08.524079 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.524043    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podStartSLOduration=12.115877305 podStartE2EDuration="35.524031474s" totalImagesPullingTime="19.24884172s" totalInitContainerRuntime="4.159312449s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:08.523350129 +0000 UTC m=+649.733248282" watchObservedRunningTime="2026-09-17 18:40:08.524031474 +0000 UTC m=+649.733929623"
Sep 17 18:40:08.545967 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.545933    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podStartSLOduration=12.387329117 podStartE2EDuration="36.5459216s" totalImagesPullingTime="20.031745293s" totalInitContainerRuntime="4.12684719s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:08.544508239 +0000 UTC m=+649.754406392" watchObservedRunningTime="2026-09-17 18:40:08.5459216 +0000 UTC m=+649.755819751"
Sep 17 18:40:08.568675 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:08.568634    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podStartSLOduration=2.81107636 podStartE2EDuration="35.568621913s" totalImagesPullingTime="28.676085843s" totalInitContainerRuntime="4.08145971s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:08.56666194 +0000 UTC m=+649.776560091" watchObservedRunningTime="2026-09-17 18:40:08.568621913 +0000 UTC m=+649.778520108"
Sep 17 18:40:09.492250 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:09.492200    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 18:40:09.492687 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:09.492252    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:40:09.493472 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:09.493355    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:40:09.493472 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:09.493442    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:40:09.497749 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:09.497697    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 18:40:09.497749 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:09.497729    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:40:10.495757 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:10.495714    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 18:40:10.496254 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:10.495844    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:40:14.496695 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:14.496665    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:40:14.497249 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:14.497217    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:40:14.497399 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:14.497235    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:40:14.497721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:14.497699    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 18:40:15.500732 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:15.500702    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:40:15.501147 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:15.500819    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:40:15.501407 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:15.501374    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 18:40:15.501510 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:15.501383    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:40:24.498261 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:24.498229    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:40:24.498688 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:24.498228    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 18:40:25.502355 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:25.502321    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 18:40:25.502746 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:25.502322    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:40:34.497930 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:34.497890    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:40:34.498398 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:34.497890    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 18:40:35.501438 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:35.501396    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:40:35.501813 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:35.501396    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 18:40:44.497321 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:44.497265    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:40:44.497765 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:44.497668    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 18:40:45.501395 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:45.501360    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:40:45.501759 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:45.501360    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 18:40:54.497976 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:54.497933    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:40:54.498470 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:54.498451    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:40:55.501931 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:55.501891    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 18:40:55.502337 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:40:55.501891    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:41:04.498271 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:04.498197    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:41:05.501741 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:05.501699    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 18:41:05.502119 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:05.501699    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:41:14.497876 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:14.497836    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:41:15.502036 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:15.501999    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:41:15.502444 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:15.502414    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:41:24.497483 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:24.497454    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:41:25.501943 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:25.501915    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:41:26.936368 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:26.934117    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"]
Sep 17 18:41:26.936368 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:26.934538    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container" containerID="cri-o://b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08" gracePeriod=30
Sep 17 18:41:26.936368 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:26.934872    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kube-rbac-proxy" containerID="cri-o://c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e" gracePeriod=30
Sep 17 18:41:27.090257 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.090225    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"]
Sep 17 18:41:27.092510 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.092491    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.097864 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.097841    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c5f0c-predictor-serving-cert\""
Sep 17 18:41:27.097950 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.097921    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c5f0c-kube-rbac-proxy-sar-config\""
Sep 17 18:41:27.108624 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.108598    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"]
Sep 17 18:41:27.215207 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.215143    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") " pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.215207 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.215175    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-proxy-tls\") pod \"success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") " pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.215207 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.215201    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sg6z\" (UniqueName: \"kubernetes.io/projected/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-api-access-5sg6z\") pod \"success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") " pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.315900 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.315872    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") " pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.316050 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.315907    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-proxy-tls\") pod \"success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") " pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.316104 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.316045    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sg6z\" (UniqueName: \"kubernetes.io/projected/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-api-access-5sg6z\") pod \"success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") " pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.316674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.316655    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") " pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.318282 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.318263    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-proxy-tls\") pod \"success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") " pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.325150 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.325119    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sg6z\" (UniqueName: \"kubernetes.io/projected/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-api-access-5sg6z\") pod \"success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") " pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.401998 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.401969    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.671792 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.671769    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"]
Sep 17 18:41:27.673371 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:41:27.673344    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf2883c_2dd9_4e71_8c96_ce7ed86a1767.slice/crio-61212a757c4c4539ce94f8715985819e5c345a5c4e9311c8535bcb6900f5f7e9 WatchSource:0}: Error finding container 61212a757c4c4539ce94f8715985819e5c345a5c4e9311c8535bcb6900f5f7e9: Status 404 returned error can't find the container with id 61212a757c4c4539ce94f8715985819e5c345a5c4e9311c8535bcb6900f5f7e9
Sep 17 18:41:27.785995 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.785970    2552 generic.go:356] "Generic (PLEG): container finished" podID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerID="c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e" exitCode=2
Sep 17 18:41:27.786095 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.786044    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" event={"ID":"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb","Type":"ContainerDied","Data":"c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e"}
Sep 17 18:41:27.787734 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.787717    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" event={"ID":"acf2883c-2dd9-4e71-8c96-ce7ed86a1767","Type":"ContainerStarted","Data":"e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691"}
Sep 17 18:41:27.787811 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.787757    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" event={"ID":"acf2883c-2dd9-4e71-8c96-ce7ed86a1767","Type":"ContainerStarted","Data":"ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b"}
Sep 17 18:41:27.787811 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.787781    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" event={"ID":"acf2883c-2dd9-4e71-8c96-ce7ed86a1767","Type":"ContainerStarted","Data":"61212a757c4c4539ce94f8715985819e5c345a5c4e9311c8535bcb6900f5f7e9"}
Sep 17 18:41:27.787811 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.787804    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.787962 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.787825    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:27.789071 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.789051    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 18:41:27.808520 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:27.808479    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podStartSLOduration=0.808467031 podStartE2EDuration="808.467031ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:27.807083361 +0000 UTC m=+729.016981514" watchObservedRunningTime="2026-09-17 18:41:27.808467031 +0000 UTC m=+729.018365185"
Sep 17 18:41:28.792024 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:28.791984    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 18:41:29.493069 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:29.493023    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Sep 17 18:41:29.977051 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:29.977029    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:41:30.139037 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.138999    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-rbac-proxy-sar-config\") pod \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") "
Sep 17 18:41:30.139037 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.139044    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg78b\" (UniqueName: \"kubernetes.io/projected/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-api-access-vg78b\") pod \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") "
Sep 17 18:41:30.139268 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.139070    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-proxy-tls\") pod \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\" (UID: \"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb\") "
Sep 17 18:41:30.139456 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.139426    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" (UID: "27e331d9-d88e-4ede-aaaa-4cb67fb35bcb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:30.141193 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.141164    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" (UID: "27e331d9-d88e-4ede-aaaa-4cb67fb35bcb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:30.141286 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.141208    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-api-access-vg78b" (OuterVolumeSpecName: "kube-api-access-vg78b") pod "27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" (UID: "27e331d9-d88e-4ede-aaaa-4cb67fb35bcb"). InnerVolumeSpecName "kube-api-access-vg78b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:30.240000 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.239969    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.240000 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.239999    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg78b\" (UniqueName: \"kubernetes.io/projected/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-kube-api-access-vg78b\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.240000 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.240009    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:41:30.800706 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.800671    2552 generic.go:356] "Generic (PLEG): container finished" podID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerID="b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08" exitCode=0
Sep 17 18:41:30.800872 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.800721    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"
Sep 17 18:41:30.800872 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.800748    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" event={"ID":"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb","Type":"ContainerDied","Data":"b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08"}
Sep 17 18:41:30.800872 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.800800    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8" event={"ID":"27e331d9-d88e-4ede-aaaa-4cb67fb35bcb","Type":"ContainerDied","Data":"4ff9ce29ce36d378436cf4193ec9dd74c270d7a45be0d5478595107147dced9f"}
Sep 17 18:41:30.800872 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.800828    2552 scope.go:117] "RemoveContainer" containerID="c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e"
Sep 17 18:41:30.808924 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.808910    2552 scope.go:117] "RemoveContainer" containerID="b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08"
Sep 17 18:41:30.815741 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.815722    2552 scope.go:117] "RemoveContainer" containerID="c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e"
Sep 17 18:41:30.815993 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:41:30.815976    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e\": container with ID starting with c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e not found: ID does not exist" containerID="c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e"
Sep 17 18:41:30.816034 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.815999    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e"} err="failed to get container status \"c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e\": rpc error: code = NotFound desc = could not find container \"c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e\": container with ID starting with c139e7e3689cb87a3e2c82bd6830269194fbe51c1633bc8ed0027c9b7a4a6b6e not found: ID does not exist"
Sep 17 18:41:30.816034 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.816026    2552 scope.go:117] "RemoveContainer" containerID="b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08"
Sep 17 18:41:30.816258 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:41:30.816244    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08\": container with ID starting with b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08 not found: ID does not exist" containerID="b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08"
Sep 17 18:41:30.816296 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.816263    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08"} err="failed to get container status \"b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08\": rpc error: code = NotFound desc = could not find container \"b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08\": container with ID starting with b036eed9aaca1e31c308547409ac4d072e26cd67abe568cf70ed9f4481f84c08 not found: ID does not exist"
Sep 17 18:41:30.822393 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.822371    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"]
Sep 17 18:41:30.826136 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:30.826118    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e0a45-predictor-dd9f8465d-ctpz8"]
Sep 17 18:41:31.063277 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:31.063186    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" path="/var/lib/kubelet/pods/27e331d9-d88e-4ede-aaaa-4cb67fb35bcb/volumes"
Sep 17 18:41:33.796348 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:33.796320    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:41:33.796858 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:33.796828    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 18:41:43.797200 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:43.797156    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 18:41:53.797657 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:41:53.797620    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 18:42:03.249045 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.249012    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"]
Sep 17 18:42:03.249554 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.249260    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" containerID="cri-o://2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9" gracePeriod=30
Sep 17 18:42:03.249554 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.249372    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kube-rbac-proxy" containerID="cri-o://2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659" gracePeriod=30
Sep 17 18:42:03.295641 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.295615    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"]
Sep 17 18:42:03.295896 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.295869    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kube-rbac-proxy" containerID="cri-o://61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37" gracePeriod=30
Sep 17 18:42:03.295992 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.295856    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" containerID="cri-o://64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517" gracePeriod=30
Sep 17 18:42:03.335896 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.335871    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"]
Sep 17 18:42:03.336134 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.336109    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" containerID="cri-o://a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8" gracePeriod=30
Sep 17 18:42:03.336219 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.336140    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kube-rbac-proxy" containerID="cri-o://817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6" gracePeriod=30
Sep 17 18:42:03.421703 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.421674    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"]
Sep 17 18:42:03.421981 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.421969    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kube-rbac-proxy"
Sep 17 18:42:03.422026 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.421982    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kube-rbac-proxy"
Sep 17 18:42:03.422026 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.421992    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container"
Sep 17 18:42:03.422026 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.421998    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container"
Sep 17 18:42:03.422121 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.422047    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kserve-container"
Sep 17 18:42:03.422121 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.422055    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e331d9-d88e-4ede-aaaa-4cb67fb35bcb" containerName="kube-rbac-proxy"
Sep 17 18:42:03.424112 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.424092    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:03.426966 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.426938    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-65928-predictor-serving-cert\""
Sep 17 18:42:03.427072 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.426941    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-65928-kube-rbac-proxy-sar-config\""
Sep 17 18:42:03.435479 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.435460    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"]
Sep 17 18:42:03.482452 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.482421    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xxx6\" (UniqueName: \"kubernetes.io/projected/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-api-access-2xxx6\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:03.482568 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.482462    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:03.482568 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.482500    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/592f6b46-9df5-4b06-beea-ac6855eb5c14-proxy-tls\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:03.583366 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.583276    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xxx6\" (UniqueName: \"kubernetes.io/projected/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-api-access-2xxx6\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:03.583366 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.583342    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:03.583564 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.583386    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/592f6b46-9df5-4b06-beea-ac6855eb5c14-proxy-tls\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:03.583564 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:03.583484    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-65928-predictor-serving-cert: secret "success-200-isvc-65928-predictor-serving-cert" not found
Sep 17 18:42:03.583564 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:03.583551    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/592f6b46-9df5-4b06-beea-ac6855eb5c14-proxy-tls podName:592f6b46-9df5-4b06-beea-ac6855eb5c14 nodeName:}" failed. No retries permitted until 2026-09-17 18:42:04.083530445 +0000 UTC m=+765.293428588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/592f6b46-9df5-4b06-beea-ac6855eb5c14-proxy-tls") pod "success-200-isvc-65928-predictor-8d4457c6-thkmw" (UID: "592f6b46-9df5-4b06-beea-ac6855eb5c14") : secret "success-200-isvc-65928-predictor-serving-cert" not found
Sep 17 18:42:03.583988 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.583963    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:03.592135 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.592113    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xxx6\" (UniqueName: \"kubernetes.io/projected/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-api-access-2xxx6\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:03.797571 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.797534    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 18:42:03.931165 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.931137    2552 generic.go:356] "Generic (PLEG): container finished" podID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerID="2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659" exitCode=2
Sep 17 18:42:03.931355 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.931217    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" event={"ID":"60a04390-4d7a-4489-bcde-f9d4161d3174","Type":"ContainerDied","Data":"2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659"}
Sep 17 18:42:03.933276 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.933257    2552 generic.go:356] "Generic (PLEG): container finished" podID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerID="61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37" exitCode=2
Sep 17 18:42:03.933400 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.933335    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" event={"ID":"ad47c559-8e84-43f5-a715-f5fd2acae340","Type":"ContainerDied","Data":"61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37"}
Sep 17 18:42:03.935228 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.935210    2552 generic.go:356] "Generic (PLEG): container finished" podID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerID="817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6" exitCode=2
Sep 17 18:42:03.935335 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:03.935241    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" event={"ID":"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d","Type":"ContainerDied","Data":"817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6"}
Sep 17 18:42:04.087846 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.087812    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/592f6b46-9df5-4b06-beea-ac6855eb5c14-proxy-tls\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:04.090168 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.090142    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/592f6b46-9df5-4b06-beea-ac6855eb5c14-proxy-tls\") pod \"success-200-isvc-65928-predictor-8d4457c6-thkmw\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") " pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:04.334264 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.334235    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:04.492913 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.492873    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Sep 17 18:42:04.497265 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.497243    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 17 18:42:04.604074 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.604049    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"]
Sep 17 18:42:04.606259 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:42:04.606225    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592f6b46_9df5_4b06_beea_ac6855eb5c14.slice/crio-902d06bd68b033b29498cdd30e14f6f6873220f3984a571d2c926688ccfd7371 WatchSource:0}: Error finding container 902d06bd68b033b29498cdd30e14f6f6873220f3984a571d2c926688ccfd7371: Status 404 returned error can't find the container with id 902d06bd68b033b29498cdd30e14f6f6873220f3984a571d2c926688ccfd7371
Sep 17 18:42:04.940579 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.940497    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" event={"ID":"592f6b46-9df5-4b06-beea-ac6855eb5c14","Type":"ContainerStarted","Data":"74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da"}
Sep 17 18:42:04.940579 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.940563    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" event={"ID":"592f6b46-9df5-4b06-beea-ac6855eb5c14","Type":"ContainerStarted","Data":"b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105"}
Sep 17 18:42:04.940738 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.940594    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" event={"ID":"592f6b46-9df5-4b06-beea-ac6855eb5c14","Type":"ContainerStarted","Data":"902d06bd68b033b29498cdd30e14f6f6873220f3984a571d2c926688ccfd7371"}
Sep 17 18:42:04.940738 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.940639    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:04.958574 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:04.958519    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" podStartSLOduration=1.9585009709999999 podStartE2EDuration="1.958500971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:04.958172644 +0000 UTC m=+766.168070797" watchObservedRunningTime="2026-09-17 18:42:04.958500971 +0000 UTC m=+766.168399123"
Sep 17 18:42:05.496878 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:05.496837    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 17 18:42:05.497348 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:05.496837    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Sep 17 18:42:05.501377 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:05.501347    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 18:42:05.501463 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:05.501347    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 18:42:05.944117 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:05.944089    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:05.945314 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:05.945269    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 18:42:06.948465 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:06.948404    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 18:42:08.188998 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.188975    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:42:08.231533 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.231511    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:42:08.321296 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321270    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad47c559-8e84-43f5-a715-f5fd2acae340-proxy-tls\") pod \"ad47c559-8e84-43f5-a715-f5fd2acae340\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") "
Sep 17 18:42:08.321441 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321320    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-rbac-proxy-sar-config\") pod \"ad47c559-8e84-43f5-a715-f5fd2acae340\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") "
Sep 17 18:42:08.321441 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321350    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad47c559-8e84-43f5-a715-f5fd2acae340-kserve-provision-location\") pod \"ad47c559-8e84-43f5-a715-f5fd2acae340\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") "
Sep 17 18:42:08.321441 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321366    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-proxy-tls\") pod \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") "
Sep 17 18:42:08.321441 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321387    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-cabundle-cert\") pod \"ad47c559-8e84-43f5-a715-f5fd2acae340\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") "
Sep 17 18:42:08.321441 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321405    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nkrx\" (UniqueName: \"kubernetes.io/projected/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-api-access-2nkrx\") pod \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") "
Sep 17 18:42:08.321678 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321442    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-rbac-proxy-sar-config\") pod \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") "
Sep 17 18:42:08.321678 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321473    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-cabundle-cert\") pod \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") "
Sep 17 18:42:08.321678 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321503    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmpjv\" (UniqueName: \"kubernetes.io/projected/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-api-access-xmpjv\") pod \"ad47c559-8e84-43f5-a715-f5fd2acae340\" (UID: \"ad47c559-8e84-43f5-a715-f5fd2acae340\") "
Sep 17 18:42:08.321678 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321533    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kserve-provision-location\") pod \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\" (UID: \"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d\") "
Sep 17 18:42:08.321873 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321718    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad47c559-8e84-43f5-a715-f5fd2acae340-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad47c559-8e84-43f5-a715-f5fd2acae340" (UID: "ad47c559-8e84-43f5-a715-f5fd2acae340"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:08.321873 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321763    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad47c559-8e84-43f5-a715-f5fd2acae340" (UID: "ad47c559-8e84-43f5-a715-f5fd2acae340"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:08.321873 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321816    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad47c559-8e84-43f5-a715-f5fd2acae340" (UID: "ad47c559-8e84-43f5-a715-f5fd2acae340"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:08.322025 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.321965    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" (UID: "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:08.322160 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.322137    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" (UID: "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:08.322160 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.322137    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" (UID: "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:08.323699 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.323680    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-api-access-xmpjv" (OuterVolumeSpecName: "kube-api-access-xmpjv") pod "ad47c559-8e84-43f5-a715-f5fd2acae340" (UID: "ad47c559-8e84-43f5-a715-f5fd2acae340"). InnerVolumeSpecName "kube-api-access-xmpjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:08.323782 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.323755    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-api-access-2nkrx" (OuterVolumeSpecName: "kube-api-access-2nkrx") pod "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" (UID: "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d"). InnerVolumeSpecName "kube-api-access-2nkrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:08.323903 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.323887    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad47c559-8e84-43f5-a715-f5fd2acae340-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad47c559-8e84-43f5-a715-f5fd2acae340" (UID: "ad47c559-8e84-43f5-a715-f5fd2acae340"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:08.323903 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.323896    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" (UID: "5a5d0f1b-a649-4c94-a4fe-a6d8103be06d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:08.422771 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422747    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.422771 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422769    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-cabundle-cert\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.422964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422783    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmpjv\" (UniqueName: \"kubernetes.io/projected/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-api-access-xmpjv\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.422964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422796    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kserve-provision-location\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.422964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422809    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad47c559-8e84-43f5-a715-f5fd2acae340-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.422964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422821    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.422964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422832    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad47c559-8e84-43f5-a715-f5fd2acae340-kserve-provision-location\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.422964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422844    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.422964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422856    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad47c559-8e84-43f5-a715-f5fd2acae340-cabundle-cert\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.422964 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.422867    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nkrx\" (UniqueName: \"kubernetes.io/projected/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d-kube-api-access-2nkrx\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:08.957845 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.957811    2552 generic.go:356] "Generic (PLEG): container finished" podID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerID="64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517" exitCode=0
Sep 17 18:42:08.958004 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.957857    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"
Sep 17 18:42:08.958004 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.957909    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" event={"ID":"ad47c559-8e84-43f5-a715-f5fd2acae340","Type":"ContainerDied","Data":"64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517"}
Sep 17 18:42:08.958004 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.957990    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf" event={"ID":"ad47c559-8e84-43f5-a715-f5fd2acae340","Type":"ContainerDied","Data":"ced9771637b44270234f98ecaaf4f3302cb23e689974396859f0d4911082180c"}
Sep 17 18:42:08.958153 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.958034    2552 scope.go:117] "RemoveContainer" containerID="61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37"
Sep 17 18:42:08.960536 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.960511    2552 generic.go:356] "Generic (PLEG): container finished" podID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerID="a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8" exitCode=0
Sep 17 18:42:08.960652 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.960602    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" event={"ID":"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d","Type":"ContainerDied","Data":"a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8"}
Sep 17 18:42:08.960768 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.960653    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf" event={"ID":"5a5d0f1b-a649-4c94-a4fe-a6d8103be06d","Type":"ContainerDied","Data":"ca6a86ed1459f84be31b928f729745c2613e7e67d851f82f25405765b33b0bda"}
Sep 17 18:42:08.960861 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.960845    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"
Sep 17 18:42:08.975591 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.975575    2552 scope.go:117] "RemoveContainer" containerID="64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517"
Sep 17 18:42:08.983674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.983653    2552 scope.go:117] "RemoveContainer" containerID="3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df"
Sep 17 18:42:08.988535 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.988510    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"]
Sep 17 18:42:08.992211 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.992190    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6586fbcf75-8m8lf"]
Sep 17 18:42:08.992465 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.992448    2552 scope.go:117] "RemoveContainer" containerID="61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37"
Sep 17 18:42:08.992893 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:08.992758    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37\": container with ID starting with 61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37 not found: ID does not exist" containerID="61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37"
Sep 17 18:42:08.992893 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.992799    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37"} err="failed to get container status \"61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37\": rpc error: code = NotFound desc = could not find container \"61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37\": container with ID starting with 61760f8d9ea0cc149faa7b52aa2d19d31c23ee338dd529423a8b6d0fbe601a37 not found: ID does not exist"
Sep 17 18:42:08.992893 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.992822    2552 scope.go:117] "RemoveContainer" containerID="64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517"
Sep 17 18:42:08.993168 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:08.993131    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517\": container with ID starting with 64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517 not found: ID does not exist" containerID="64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517"
Sep 17 18:42:08.993287 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.993175    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517"} err="failed to get container status \"64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517\": rpc error: code = NotFound desc = could not find container \"64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517\": container with ID starting with 64ed28a5cbb16009e28c81945c0500547f6f764902414ecf31b81813f0f72517 not found: ID does not exist"
Sep 17 18:42:08.993287 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.993192    2552 scope.go:117] "RemoveContainer" containerID="3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df"
Sep 17 18:42:08.993535 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:08.993511    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df\": container with ID starting with 3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df not found: ID does not exist" containerID="3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df"
Sep 17 18:42:08.993602 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.993543    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df"} err="failed to get container status \"3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df\": rpc error: code = NotFound desc = could not find container \"3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df\": container with ID starting with 3de9273f0b3ac34bdff65b7e12268dc29d6b9bc501884cfb198c57acfdaf01df not found: ID does not exist"
Sep 17 18:42:08.993602 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:08.993564    2552 scope.go:117] "RemoveContainer" containerID="817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6"
Sep 17 18:42:09.005041 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.005015    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"]
Sep 17 18:42:09.005417 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.005399    2552 scope.go:117] "RemoveContainer" containerID="a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8"
Sep 17 18:42:09.008341 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.008320    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-n9npf"]
Sep 17 18:42:09.013729 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.013711    2552 scope.go:117] "RemoveContainer" containerID="077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2"
Sep 17 18:42:09.021206 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.021183    2552 scope.go:117] "RemoveContainer" containerID="817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6"
Sep 17 18:42:09.021498 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:09.021474    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6\": container with ID starting with 817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6 not found: ID does not exist" containerID="817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6"
Sep 17 18:42:09.021588 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.021506    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6"} err="failed to get container status \"817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6\": rpc error: code = NotFound desc = could not find container \"817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6\": container with ID starting with 817c109b382daa847eea250ceba2947c3bbfc6f4c11cc4c3021bb42282899bf6 not found: ID does not exist"
Sep 17 18:42:09.021588 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.021530    2552 scope.go:117] "RemoveContainer" containerID="a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8"
Sep 17 18:42:09.021821 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:09.021799    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8\": container with ID starting with a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8 not found: ID does not exist" containerID="a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8"
Sep 17 18:42:09.021865 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.021824    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8"} err="failed to get container status \"a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8\": rpc error: code = NotFound desc = could not find container \"a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8\": container with ID starting with a6edbd18a343890796daadaf6d679b317203c1e81dcc546f336f27b9a66bd7e8 not found: ID does not exist"
Sep 17 18:42:09.021865 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.021845    2552 scope.go:117] "RemoveContainer" containerID="077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2"
Sep 17 18:42:09.022136 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:09.022111    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2\": container with ID starting with 077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2 not found: ID does not exist" containerID="077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2"
Sep 17 18:42:09.022179 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.022145    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2"} err="failed to get container status \"077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2\": rpc error: code = NotFound desc = could not find container \"077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2\": container with ID starting with 077f6d8b836a2fcb81579923520016c476214a04a0c0d4511fc2f3913a536be2 not found: ID does not exist"
Sep 17 18:42:09.063279 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.063254    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" path="/var/lib/kubelet/pods/5a5d0f1b-a649-4c94-a4fe-a6d8103be06d/volumes"
Sep 17 18:42:09.063764 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.063749    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" path="/var/lib/kubelet/pods/ad47c559-8e84-43f5-a715-f5fd2acae340/volumes"
Sep 17 18:42:09.098291 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.098273    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:42:09.228616 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.228550    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-rbac-proxy-sar-config\") pod \"60a04390-4d7a-4489-bcde-f9d4161d3174\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") "
Sep 17 18:42:09.228616 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.228584    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbb68\" (UniqueName: \"kubernetes.io/projected/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-api-access-fbb68\") pod \"60a04390-4d7a-4489-bcde-f9d4161d3174\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") "
Sep 17 18:42:09.229046 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.228621    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a04390-4d7a-4489-bcde-f9d4161d3174-proxy-tls\") pod \"60a04390-4d7a-4489-bcde-f9d4161d3174\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") "
Sep 17 18:42:09.229046 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.228653    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a04390-4d7a-4489-bcde-f9d4161d3174-kserve-provision-location\") pod \"60a04390-4d7a-4489-bcde-f9d4161d3174\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") "
Sep 17 18:42:09.229046 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.228677    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-cabundle-cert\") pod \"60a04390-4d7a-4489-bcde-f9d4161d3174\" (UID: \"60a04390-4d7a-4489-bcde-f9d4161d3174\") "
Sep 17 18:42:09.229046 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.229021    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a04390-4d7a-4489-bcde-f9d4161d3174-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60a04390-4d7a-4489-bcde-f9d4161d3174" (UID: "60a04390-4d7a-4489-bcde-f9d4161d3174"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:09.229046 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.229034    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "60a04390-4d7a-4489-bcde-f9d4161d3174" (UID: "60a04390-4d7a-4489-bcde-f9d4161d3174"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:09.229253 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.229058    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "60a04390-4d7a-4489-bcde-f9d4161d3174" (UID: "60a04390-4d7a-4489-bcde-f9d4161d3174"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:09.230646 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.230627    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-api-access-fbb68" (OuterVolumeSpecName: "kube-api-access-fbb68") pod "60a04390-4d7a-4489-bcde-f9d4161d3174" (UID: "60a04390-4d7a-4489-bcde-f9d4161d3174"). InnerVolumeSpecName "kube-api-access-fbb68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:09.230860 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.230847    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a04390-4d7a-4489-bcde-f9d4161d3174-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "60a04390-4d7a-4489-bcde-f9d4161d3174" (UID: "60a04390-4d7a-4489-bcde-f9d4161d3174"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:09.329345 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.329317    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbb68\" (UniqueName: \"kubernetes.io/projected/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-api-access-fbb68\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:09.329345 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.329340    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60a04390-4d7a-4489-bcde-f9d4161d3174-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:09.329345 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.329349    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60a04390-4d7a-4489-bcde-f9d4161d3174-kserve-provision-location\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:09.329537 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.329361    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-cabundle-cert\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:09.329537 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.329369    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60a04390-4d7a-4489-bcde-f9d4161d3174-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:42:09.968349 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.968313    2552 generic.go:356] "Generic (PLEG): container finished" podID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerID="2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9" exitCode=0
Sep 17 18:42:09.968540 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.968376    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"
Sep 17 18:42:09.968540 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.968390    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" event={"ID":"60a04390-4d7a-4489-bcde-f9d4161d3174","Type":"ContainerDied","Data":"2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9"}
Sep 17 18:42:09.968540 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.968450    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr" event={"ID":"60a04390-4d7a-4489-bcde-f9d4161d3174","Type":"ContainerDied","Data":"5320088dd74a2b4895181524899f7333aba6f461835e50e3f42e83734eef6304"}
Sep 17 18:42:09.968540 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.968484    2552 scope.go:117] "RemoveContainer" containerID="2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659"
Sep 17 18:42:09.976286 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.976266    2552 scope.go:117] "RemoveContainer" containerID="2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9"
Sep 17 18:42:09.983017 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.983001    2552 scope.go:117] "RemoveContainer" containerID="743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f"
Sep 17 18:42:09.989430 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.989412    2552 scope.go:117] "RemoveContainer" containerID="2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659"
Sep 17 18:42:09.989692 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:09.989672    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659\": container with ID starting with 2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659 not found: ID does not exist" containerID="2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659"
Sep 17 18:42:09.989741 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.989699    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659"} err="failed to get container status \"2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659\": rpc error: code = NotFound desc = could not find container \"2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659\": container with ID starting with 2fd83b902a3ae0586fd1c628a16f8781fe1718f825f37dd48f6b76c805048659 not found: ID does not exist"
Sep 17 18:42:09.989741 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.989714    2552 scope.go:117] "RemoveContainer" containerID="2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9"
Sep 17 18:42:09.989970 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:09.989953    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9\": container with ID starting with 2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9 not found: ID does not exist" containerID="2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9"
Sep 17 18:42:09.990014 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.989975    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9"} err="failed to get container status \"2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9\": rpc error: code = NotFound desc = could not find container \"2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9\": container with ID starting with 2d855bd4c7f627f4ad622bcb62902ab5d8a34909cc798267208941b0ba3e7cd9 not found: ID does not exist"
Sep 17 18:42:09.990014 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.989991    2552 scope.go:117] "RemoveContainer" containerID="743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f"
Sep 17 18:42:09.990235 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:42:09.990219    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f\": container with ID starting with 743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f not found: ID does not exist" containerID="743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f"
Sep 17 18:42:09.990277 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.990240    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f"} err="failed to get container status \"743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f\": rpc error: code = NotFound desc = could not find container \"743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f\": container with ID starting with 743c7633d538dad1eb8c09e67081f8f87a02d81a0ce5d812179e40ccd87e093f not found: ID does not exist"
Sep 17 18:42:09.994494 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.994476    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"]
Sep 17 18:42:09.999227 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:09.999208    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bb8cf688-7vghr"]
Sep 17 18:42:11.062794 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:11.062755    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" path="/var/lib/kubelet/pods/60a04390-4d7a-4489-bcde-f9d4161d3174/volumes"
Sep 17 18:42:11.952315 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:11.952272    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:42:11.952805 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:11.952775    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 18:42:13.798004 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:13.797974    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:42:21.953181 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:21.953142    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 18:42:31.953248 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:31.953210    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 18:42:41.953011 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:41.952916    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 18:42:51.953467 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:42:51.953436    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:44:20.990114 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:44:20.990037    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:44:20.995959 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:44:20.995940    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:49:21.013405 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:49:21.013378    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:49:21.017110 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:49:21.017090    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:50:41.841766 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.841687    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"]
Sep 17 18:50:41.842383 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.842000    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container" containerID="cri-o://ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b" gracePeriod=30
Sep 17 18:50:41.842383 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.842033    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kube-rbac-proxy" containerID="cri-o://e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691" gracePeriod=30
Sep 17 18:50:41.979440 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979415    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"]
Sep 17 18:50:41.979674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979664    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kube-rbac-proxy"
Sep 17 18:50:41.979721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979676    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kube-rbac-proxy"
Sep 17 18:50:41.979721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979686    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kube-rbac-proxy"
Sep 17 18:50:41.979721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979692    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kube-rbac-proxy"
Sep 17 18:50:41.979721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979700    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kube-rbac-proxy"
Sep 17 18:50:41.979721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979706    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kube-rbac-proxy"
Sep 17 18:50:41.979721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979711    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container"
Sep 17 18:50:41.979721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979716    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container"
Sep 17 18:50:41.979721 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979722    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="storage-initializer"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979727    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="storage-initializer"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979735    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="storage-initializer"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979740    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="storage-initializer"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979746    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="storage-initializer"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979751    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="storage-initializer"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979759    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979764    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979776    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979781    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979832    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kserve-container"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979842    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kserve-container"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979851    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kube-rbac-proxy"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979859    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad47c559-8e84-43f5-a715-f5fd2acae340" containerName="kserve-container"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979867    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a5d0f1b-a649-4c94-a4fe-a6d8103be06d" containerName="kube-rbac-proxy"
Sep 17 18:50:41.979973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.979876    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a04390-4d7a-4489-bcde-f9d4161d3174" containerName="kube-rbac-proxy"
Sep 17 18:50:41.982610 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.982594    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:41.985805 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.985782    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c1827-predictor-serving-cert\""
Sep 17 18:50:41.988230 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:41.988211    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c1827-kube-rbac-proxy-sar-config\""
Sep 17 18:50:42.004724 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.004700    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"]
Sep 17 18:50:42.071632 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.071607    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.071763 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.071646    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-proxy-tls\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.071763 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.071668    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdbb\" (UniqueName: \"kubernetes.io/projected/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-api-access-qfdbb\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.172994 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.172911    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.172994 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.172960    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-proxy-tls\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.172994 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.172983    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdbb\" (UniqueName: \"kubernetes.io/projected/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-api-access-qfdbb\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.173244 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:50:42.173099    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-c1827-predictor-serving-cert: secret "success-200-isvc-c1827-predictor-serving-cert" not found
Sep 17 18:50:42.173244 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:50:42.173169    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-proxy-tls podName:12c747a1-b5ef-44d2-9cf1-dc75c416eab4 nodeName:}" failed. No retries permitted until 2026-09-17 18:50:42.673148749 +0000 UTC m=+1283.883046886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-proxy-tls") pod "success-200-isvc-c1827-predictor-64748d747d-75nl8" (UID: "12c747a1-b5ef-44d2-9cf1-dc75c416eab4") : secret "success-200-isvc-c1827-predictor-serving-cert" not found
Sep 17 18:50:42.173750 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.173731    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.184046 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.184022    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdbb\" (UniqueName: \"kubernetes.io/projected/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-api-access-qfdbb\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.676644 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.676616    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-proxy-tls\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.678932 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.678905    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-proxy-tls\") pod \"success-200-isvc-c1827-predictor-64748d747d-75nl8\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") " pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:42.698719 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.698693    2552 generic.go:356] "Generic (PLEG): container finished" podID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerID="e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691" exitCode=2
Sep 17 18:50:42.698828 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.698750    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" event={"ID":"acf2883c-2dd9-4e71-8c96-ce7ed86a1767","Type":"ContainerDied","Data":"e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691"}
Sep 17 18:50:42.893713 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:42.893667    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:43.160451 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:43.160422    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"]
Sep 17 18:50:43.162922 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:50:43.162888    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c747a1_b5ef_44d2_9cf1_dc75c416eab4.slice/crio-03f595eb85d69ea42cf3d3eb8e38f53715de736a37766af4be3fa4250a81f3b5 WatchSource:0}: Error finding container 03f595eb85d69ea42cf3d3eb8e38f53715de736a37766af4be3fa4250a81f3b5: Status 404 returned error can't find the container with id 03f595eb85d69ea42cf3d3eb8e38f53715de736a37766af4be3fa4250a81f3b5
Sep 17 18:50:43.704321 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:43.704265    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" event={"ID":"12c747a1-b5ef-44d2-9cf1-dc75c416eab4","Type":"ContainerStarted","Data":"5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707"}
Sep 17 18:50:43.704506 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:43.704362    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" event={"ID":"12c747a1-b5ef-44d2-9cf1-dc75c416eab4","Type":"ContainerStarted","Data":"4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299"}
Sep 17 18:50:43.704506 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:43.704397    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" event={"ID":"12c747a1-b5ef-44d2-9cf1-dc75c416eab4","Type":"ContainerStarted","Data":"03f595eb85d69ea42cf3d3eb8e38f53715de736a37766af4be3fa4250a81f3b5"}
Sep 17 18:50:43.704506 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:43.704436    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:43.723552 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:43.723506    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" podStartSLOduration=2.7234902610000002 podStartE2EDuration="2.723490261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:43.722329328 +0000 UTC m=+1284.932227480" watchObservedRunningTime="2026-09-17 18:50:43.723490261 +0000 UTC m=+1284.933388412"
Sep 17 18:50:43.792917 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:43.792885    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 17 18:50:43.797217 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:43.797191    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 18:50:44.707924 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:44.707896    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:44.709092 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:44.709066    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 18:50:44.884877 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:44.884855    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:50:44.991082 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:44.991012    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sg6z\" (UniqueName: \"kubernetes.io/projected/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-api-access-5sg6z\") pod \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") "
Sep 17 18:50:44.991082 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:44.991066    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-rbac-proxy-sar-config\") pod \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") "
Sep 17 18:50:44.991234 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:44.991095    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-proxy-tls\") pod \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\" (UID: \"acf2883c-2dd9-4e71-8c96-ce7ed86a1767\") "
Sep 17 18:50:44.991511 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:44.991484    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "acf2883c-2dd9-4e71-8c96-ce7ed86a1767" (UID: "acf2883c-2dd9-4e71-8c96-ce7ed86a1767"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:50:44.992989 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:44.992962    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-api-access-5sg6z" (OuterVolumeSpecName: "kube-api-access-5sg6z") pod "acf2883c-2dd9-4e71-8c96-ce7ed86a1767" (UID: "acf2883c-2dd9-4e71-8c96-ce7ed86a1767"). InnerVolumeSpecName "kube-api-access-5sg6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:50:44.993085 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:44.993019    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "acf2883c-2dd9-4e71-8c96-ce7ed86a1767" (UID: "acf2883c-2dd9-4e71-8c96-ce7ed86a1767"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:50:45.091640 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.091611    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sg6z\" (UniqueName: \"kubernetes.io/projected/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-api-access-5sg6z\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:50:45.091640 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.091637    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:50:45.091769 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.091652    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acf2883c-2dd9-4e71-8c96-ce7ed86a1767-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:50:45.713292 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.713259    2552 generic.go:356] "Generic (PLEG): container finished" podID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerID="ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b" exitCode=0
Sep 17 18:50:45.713811 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.713327    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"
Sep 17 18:50:45.713811 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.713331    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" event={"ID":"acf2883c-2dd9-4e71-8c96-ce7ed86a1767","Type":"ContainerDied","Data":"ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b"}
Sep 17 18:50:45.713811 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.713386    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb" event={"ID":"acf2883c-2dd9-4e71-8c96-ce7ed86a1767","Type":"ContainerDied","Data":"61212a757c4c4539ce94f8715985819e5c345a5c4e9311c8535bcb6900f5f7e9"}
Sep 17 18:50:45.713811 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.713411    2552 scope.go:117] "RemoveContainer" containerID="e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691"
Sep 17 18:50:45.713973 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.713945    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 18:50:45.721167 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.721149    2552 scope.go:117] "RemoveContainer" containerID="ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b"
Sep 17 18:50:45.727892 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.727876    2552 scope.go:117] "RemoveContainer" containerID="e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691"
Sep 17 18:50:45.728152 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:50:45.728131    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691\": container with ID starting with e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691 not found: ID does not exist" containerID="e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691"
Sep 17 18:50:45.728206 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.728160    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691"} err="failed to get container status \"e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691\": rpc error: code = NotFound desc = could not find container \"e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691\": container with ID starting with e6199b1e0d7023759f87b3ffa4448fc4451056cbf0b04a12898a1a051f8f2691 not found: ID does not exist"
Sep 17 18:50:45.728206 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.728176    2552 scope.go:117] "RemoveContainer" containerID="ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b"
Sep 17 18:50:45.728437 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:50:45.728419    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b\": container with ID starting with ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b not found: ID does not exist" containerID="ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b"
Sep 17 18:50:45.728478 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.728443    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b"} err="failed to get container status \"ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b\": rpc error: code = NotFound desc = could not find container \"ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b\": container with ID starting with ec406d5cddf81cec1f7a6468fe7bc252f405baea66675337d78f1ae1136d554b not found: ID does not exist"
Sep 17 18:50:45.733679 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.733657    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"]
Sep 17 18:50:45.738358 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:45.738339    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c5f0c-predictor-78fd55bc98-r8vjb"]
Sep 17 18:50:47.063521 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:47.063491    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" path="/var/lib/kubelet/pods/acf2883c-2dd9-4e71-8c96-ce7ed86a1767/volumes"
Sep 17 18:50:50.718773 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:50.718742    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:50:50.719389 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:50:50.719144    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 18:51:00.719912 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:00.719874    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 18:51:10.719133 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:10.719095    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 18:51:18.231519 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.231490    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"]
Sep 17 18:51:18.231932 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.231697    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container" containerID="cri-o://b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105" gracePeriod=30
Sep 17 18:51:18.231932 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.231752    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kube-rbac-proxy" containerID="cri-o://74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da" gracePeriod=30
Sep 17 18:51:18.403418 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.403391    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"]
Sep 17 18:51:18.403650 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.403639    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container"
Sep 17 18:51:18.403692 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.403651    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container"
Sep 17 18:51:18.403692 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.403668    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kube-rbac-proxy"
Sep 17 18:51:18.403692 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.403674    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kube-rbac-proxy"
Sep 17 18:51:18.403781 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.403722    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kserve-container"
Sep 17 18:51:18.403781 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.403728    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf2883c-2dd9-4e71-8c96-ce7ed86a1767" containerName="kube-rbac-proxy"
Sep 17 18:51:18.406414 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.406390    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:18.414229 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.414208    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31957-kube-rbac-proxy-sar-config\""
Sep 17 18:51:18.414622 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.414595    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-31957-predictor-serving-cert\""
Sep 17 18:51:18.423199 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.423174    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"]
Sep 17 18:51:18.513497 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.513422    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:18.513497 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.513486    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcfj\" (UniqueName: \"kubernetes.io/projected/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-api-access-9fcfj\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:18.513704 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.513529    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b8e7a-1198-4d81-a8e4-992081cee5e7-proxy-tls\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:18.614829 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.614796    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcfj\" (UniqueName: \"kubernetes.io/projected/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-api-access-9fcfj\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:18.614829 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.614831    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b8e7a-1198-4d81-a8e4-992081cee5e7-proxy-tls\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:18.615052 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.614862    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:18.615052 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:51:18.615011    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-31957-predictor-serving-cert: secret "success-200-isvc-31957-predictor-serving-cert" not found
Sep 17 18:51:18.615132 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:51:18.615087    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/385b8e7a-1198-4d81-a8e4-992081cee5e7-proxy-tls podName:385b8e7a-1198-4d81-a8e4-992081cee5e7 nodeName:}" failed. No retries permitted until 2026-09-17 18:51:19.115062579 +0000 UTC m=+1320.324960715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/385b8e7a-1198-4d81-a8e4-992081cee5e7-proxy-tls") pod "success-200-isvc-31957-predictor-bb49bf486-kkdf8" (UID: "385b8e7a-1198-4d81-a8e4-992081cee5e7") : secret "success-200-isvc-31957-predictor-serving-cert" not found
Sep 17 18:51:18.615559 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.615542    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:18.625954 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.625931    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcfj\" (UniqueName: \"kubernetes.io/projected/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-api-access-9fcfj\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:18.829572 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.829486    2552 generic.go:356] "Generic (PLEG): container finished" podID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerID="74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da" exitCode=2
Sep 17 18:51:18.829572 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:18.829544    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" event={"ID":"592f6b46-9df5-4b06-beea-ac6855eb5c14","Type":"ContainerDied","Data":"74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da"}
Sep 17 18:51:19.119756 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:19.119725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b8e7a-1198-4d81-a8e4-992081cee5e7-proxy-tls\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:19.122094 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:19.122075    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b8e7a-1198-4d81-a8e4-992081cee5e7-proxy-tls\") pod \"success-200-isvc-31957-predictor-bb49bf486-kkdf8\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") " pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:19.316670 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:19.316632    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:19.595011 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:19.594987    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"]
Sep 17 18:51:19.596849 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:51:19.596822    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385b8e7a_1198_4d81_a8e4_992081cee5e7.slice/crio-98e4f5f94cfe87c3e1fa7f9fdd98dad32b342ec58fc0a0bb9d6d865f09c90fe2 WatchSource:0}: Error finding container 98e4f5f94cfe87c3e1fa7f9fdd98dad32b342ec58fc0a0bb9d6d865f09c90fe2: Status 404 returned error can't find the container with id 98e4f5f94cfe87c3e1fa7f9fdd98dad32b342ec58fc0a0bb9d6d865f09c90fe2
Sep 17 18:51:19.835045 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:19.835015    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" event={"ID":"385b8e7a-1198-4d81-a8e4-992081cee5e7","Type":"ContainerStarted","Data":"68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2"}
Sep 17 18:51:19.835198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:19.835077    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" event={"ID":"385b8e7a-1198-4d81-a8e4-992081cee5e7","Type":"ContainerStarted","Data":"6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5"}
Sep 17 18:51:19.835198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:19.835096    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" event={"ID":"385b8e7a-1198-4d81-a8e4-992081cee5e7","Type":"ContainerStarted","Data":"98e4f5f94cfe87c3e1fa7f9fdd98dad32b342ec58fc0a0bb9d6d865f09c90fe2"}
Sep 17 18:51:19.835198 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:19.835180    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:19.858452 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:19.857287    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" podStartSLOduration=1.857261826 podStartE2EDuration="1.857261826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:19.853643667 +0000 UTC m=+1321.063541811" watchObservedRunningTime="2026-09-17 18:51:19.857261826 +0000 UTC m=+1321.067159977"
Sep 17 18:51:20.720152 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:20.720113    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 18:51:20.841403 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:20.841372    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:20.842580 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:20.842557    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 18:51:21.467725 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.467702    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:51:21.540199 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.540174    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-rbac-proxy-sar-config\") pod \"592f6b46-9df5-4b06-beea-ac6855eb5c14\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") "
Sep 17 18:51:21.540355 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.540211    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xxx6\" (UniqueName: \"kubernetes.io/projected/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-api-access-2xxx6\") pod \"592f6b46-9df5-4b06-beea-ac6855eb5c14\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") "
Sep 17 18:51:21.540355 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.540249    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/592f6b46-9df5-4b06-beea-ac6855eb5c14-proxy-tls\") pod \"592f6b46-9df5-4b06-beea-ac6855eb5c14\" (UID: \"592f6b46-9df5-4b06-beea-ac6855eb5c14\") "
Sep 17 18:51:21.540634 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.540598    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "592f6b46-9df5-4b06-beea-ac6855eb5c14" (UID: "592f6b46-9df5-4b06-beea-ac6855eb5c14"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:21.542320 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.542278    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/592f6b46-9df5-4b06-beea-ac6855eb5c14-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "592f6b46-9df5-4b06-beea-ac6855eb5c14" (UID: "592f6b46-9df5-4b06-beea-ac6855eb5c14"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:51:21.542423 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.542388    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-api-access-2xxx6" (OuterVolumeSpecName: "kube-api-access-2xxx6") pod "592f6b46-9df5-4b06-beea-ac6855eb5c14" (UID: "592f6b46-9df5-4b06-beea-ac6855eb5c14"). InnerVolumeSpecName "kube-api-access-2xxx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:51:21.640920 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.640839    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:51:21.640920 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.640875    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xxx6\" (UniqueName: \"kubernetes.io/projected/592f6b46-9df5-4b06-beea-ac6855eb5c14-kube-api-access-2xxx6\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:51:21.640920 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.640885    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/592f6b46-9df5-4b06-beea-ac6855eb5c14-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:51:21.846707 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.846674    2552 generic.go:356] "Generic (PLEG): container finished" podID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerID="b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105" exitCode=0
Sep 17 18:51:21.847156 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.846724    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"
Sep 17 18:51:21.847156 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.846761    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" event={"ID":"592f6b46-9df5-4b06-beea-ac6855eb5c14","Type":"ContainerDied","Data":"b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105"}
Sep 17 18:51:21.847156 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.846826    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw" event={"ID":"592f6b46-9df5-4b06-beea-ac6855eb5c14","Type":"ContainerDied","Data":"902d06bd68b033b29498cdd30e14f6f6873220f3984a571d2c926688ccfd7371"}
Sep 17 18:51:21.847156 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.846871    2552 scope.go:117] "RemoveContainer" containerID="74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da"
Sep 17 18:51:21.847425 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.847397    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 18:51:21.854807 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.854791    2552 scope.go:117] "RemoveContainer" containerID="b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105"
Sep 17 18:51:21.863126 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.863102    2552 scope.go:117] "RemoveContainer" containerID="74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da"
Sep 17 18:51:21.863479 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:51:21.863460    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da\": container with ID starting with 74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da not found: ID does not exist" containerID="74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da"
Sep 17 18:51:21.863569 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.863487    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da"} err="failed to get container status \"74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da\": rpc error: code = NotFound desc = could not find container \"74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da\": container with ID starting with 74d24a1ca36127b0fadbbb2973d732088d8e7c76687fc51fd51a534faf1aa0da not found: ID does not exist"
Sep 17 18:51:21.863569 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.863502    2552 scope.go:117] "RemoveContainer" containerID="b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105"
Sep 17 18:51:21.863773 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:51:21.863760    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105\": container with ID starting with b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105 not found: ID does not exist" containerID="b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105"
Sep 17 18:51:21.863820 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.863776    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105"} err="failed to get container status \"b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105\": rpc error: code = NotFound desc = could not find container \"b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105\": container with ID starting with b36b78abddc715bbf22f5397fca237063ea3377bf59c309a96bf3ec9f031c105 not found: ID does not exist"
Sep 17 18:51:21.868312 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.868278    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"]
Sep 17 18:51:21.871996 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:21.871976    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65928-predictor-8d4457c6-thkmw"]
Sep 17 18:51:23.065440 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:23.065403    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" path="/var/lib/kubelet/pods/592f6b46-9df5-4b06-beea-ac6855eb5c14/volumes"
Sep 17 18:51:26.852070 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:26.852042    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:51:26.852519 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:26.852494    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 18:51:30.720298 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:30.720269    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:51:36.853515 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:36.853418    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 18:51:46.852712 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:46.852664    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 18:51:52.160836 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.160795    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"]
Sep 17 18:51:52.161276 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.161060    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container" containerID="cri-o://4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299" gracePeriod=30
Sep 17 18:51:52.161276 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.161110    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kube-rbac-proxy" containerID="cri-o://5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707" gracePeriod=30
Sep 17 18:51:52.233089 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.233051    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"]
Sep 17 18:51:52.233515 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.233491    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container"
Sep 17 18:51:52.233515 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.233514    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container"
Sep 17 18:51:52.233674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.233522    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kube-rbac-proxy"
Sep 17 18:51:52.233674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.233528    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kube-rbac-proxy"
Sep 17 18:51:52.233674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.233578    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kserve-container"
Sep 17 18:51:52.233674 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.233587    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="592f6b46-9df5-4b06-beea-ac6855eb5c14" containerName="kube-rbac-proxy"
Sep 17 18:51:52.236491 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.236475    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.239494 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.239473    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-10598-predictor-serving-cert\""
Sep 17 18:51:52.239588 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.239502    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-10598-kube-rbac-proxy-sar-config\""
Sep 17 18:51:52.252643 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.252616    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"]
Sep 17 18:51:52.358335 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.358283    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjwp\" (UniqueName: \"kubernetes.io/projected/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-api-access-4fjwp\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.358514 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.358420    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafa2187-a8f6-4883-9d2d-8731e1ba2952-proxy-tls\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.358514 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.358465    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.459531 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.459447    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.459531 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.459489    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjwp\" (UniqueName: \"kubernetes.io/projected/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-api-access-4fjwp\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.459750 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.459563    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafa2187-a8f6-4883-9d2d-8731e1ba2952-proxy-tls\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.459750 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:51:52.459677    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-10598-predictor-serving-cert: secret "success-200-isvc-10598-predictor-serving-cert" not found
Sep 17 18:51:52.459750 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:51:52.459742    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eafa2187-a8f6-4883-9d2d-8731e1ba2952-proxy-tls podName:eafa2187-a8f6-4883-9d2d-8731e1ba2952 nodeName:}" failed. No retries permitted until 2026-09-17 18:51:52.959721063 +0000 UTC m=+1354.169619194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eafa2187-a8f6-4883-9d2d-8731e1ba2952-proxy-tls") pod "success-200-isvc-10598-predictor-756678795d-kb629" (UID: "eafa2187-a8f6-4883-9d2d-8731e1ba2952") : secret "success-200-isvc-10598-predictor-serving-cert" not found
Sep 17 18:51:52.460171 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.460151    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.472768 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.472744    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjwp\" (UniqueName: \"kubernetes.io/projected/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-api-access-4fjwp\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.961926 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.961886    2552 generic.go:356] "Generic (PLEG): container finished" podID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerID="5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707" exitCode=2
Sep 17 18:51:52.962165 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.961953    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" event={"ID":"12c747a1-b5ef-44d2-9cf1-dc75c416eab4","Type":"ContainerDied","Data":"5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707"}
Sep 17 18:51:52.963363 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.963335    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafa2187-a8f6-4883-9d2d-8731e1ba2952-proxy-tls\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:52.965827 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:52.965800    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafa2187-a8f6-4883-9d2d-8731e1ba2952-proxy-tls\") pod \"success-200-isvc-10598-predictor-756678795d-kb629\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") " pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:53.150827 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:53.150791    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:53.421895 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:53.421872    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"]
Sep 17 18:51:53.424140 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:51:53.424115    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafa2187_a8f6_4883_9d2d_8731e1ba2952.slice/crio-b4c178a07cddefdd99f7bc212050d996670a2c046663a158b65b49e333eb58b1 WatchSource:0}: Error finding container b4c178a07cddefdd99f7bc212050d996670a2c046663a158b65b49e333eb58b1: Status 404 returned error can't find the container with id b4c178a07cddefdd99f7bc212050d996670a2c046663a158b65b49e333eb58b1
Sep 17 18:51:53.968209 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:53.968172    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" event={"ID":"eafa2187-a8f6-4883-9d2d-8731e1ba2952","Type":"ContainerStarted","Data":"732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062"}
Sep 17 18:51:53.968402 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:53.968237    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" event={"ID":"eafa2187-a8f6-4883-9d2d-8731e1ba2952","Type":"ContainerStarted","Data":"55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340"}
Sep 17 18:51:53.968402 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:53.968264    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" event={"ID":"eafa2187-a8f6-4883-9d2d-8731e1ba2952","Type":"ContainerStarted","Data":"b4c178a07cddefdd99f7bc212050d996670a2c046663a158b65b49e333eb58b1"}
Sep 17 18:51:53.968402 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:53.968331    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:53.968532 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:53.968487    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:53.969678 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:53.969654    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:51:53.988356 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:53.988294    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podStartSLOduration=1.988283598 podStartE2EDuration="1.988283598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:53.987060846 +0000 UTC m=+1355.196959033" watchObservedRunningTime="2026-09-17 18:51:53.988283598 +0000 UTC m=+1355.198181751"
Sep 17 18:51:54.972406 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:54.972373    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:51:55.206460 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.206439    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:51:55.278657 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.278591    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfdbb\" (UniqueName: \"kubernetes.io/projected/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-api-access-qfdbb\") pod \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") "
Sep 17 18:51:55.278657 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.278639    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-rbac-proxy-sar-config\") pod \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") "
Sep 17 18:51:55.278861 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.278666    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-proxy-tls\") pod \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\" (UID: \"12c747a1-b5ef-44d2-9cf1-dc75c416eab4\") "
Sep 17 18:51:55.278996 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.278974    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12c747a1-b5ef-44d2-9cf1-dc75c416eab4" (UID: "12c747a1-b5ef-44d2-9cf1-dc75c416eab4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:55.280695 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.280670    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12c747a1-b5ef-44d2-9cf1-dc75c416eab4" (UID: "12c747a1-b5ef-44d2-9cf1-dc75c416eab4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:51:55.280807 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.280714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-api-access-qfdbb" (OuterVolumeSpecName: "kube-api-access-qfdbb") pod "12c747a1-b5ef-44d2-9cf1-dc75c416eab4" (UID: "12c747a1-b5ef-44d2-9cf1-dc75c416eab4"). InnerVolumeSpecName "kube-api-access-qfdbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:51:55.379088 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.379052    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:51:55.379088 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.379085    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdbb\" (UniqueName: \"kubernetes.io/projected/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-api-access-qfdbb\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:51:55.379088 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.379096    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c747a1-b5ef-44d2-9cf1-dc75c416eab4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:51:55.976689 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.976654    2552 generic.go:356] "Generic (PLEG): container finished" podID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerID="4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299" exitCode=0
Sep 17 18:51:55.977099 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.976702    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"
Sep 17 18:51:55.977099 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.976735    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" event={"ID":"12c747a1-b5ef-44d2-9cf1-dc75c416eab4","Type":"ContainerDied","Data":"4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299"}
Sep 17 18:51:55.977099 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.976791    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8" event={"ID":"12c747a1-b5ef-44d2-9cf1-dc75c416eab4","Type":"ContainerDied","Data":"03f595eb85d69ea42cf3d3eb8e38f53715de736a37766af4be3fa4250a81f3b5"}
Sep 17 18:51:55.977099 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.976817    2552 scope.go:117] "RemoveContainer" containerID="5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707"
Sep 17 18:51:55.985359 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.985338    2552 scope.go:117] "RemoveContainer" containerID="4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299"
Sep 17 18:51:55.992667 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.992651    2552 scope.go:117] "RemoveContainer" containerID="5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707"
Sep 17 18:51:55.992928 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:51:55.992910    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707\": container with ID starting with 5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707 not found: ID does not exist" containerID="5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707"
Sep 17 18:51:55.992971 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.992936    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707"} err="failed to get container status \"5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707\": rpc error: code = NotFound desc = could not find container \"5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707\": container with ID starting with 5b1d0aeac0e561392120eaa613ed400085783590667a8cdd67c470e056f12707 not found: ID does not exist"
Sep 17 18:51:55.992971 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.992952    2552 scope.go:117] "RemoveContainer" containerID="4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299"
Sep 17 18:51:55.993201 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:51:55.993185    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299\": container with ID starting with 4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299 not found: ID does not exist" containerID="4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299"
Sep 17 18:51:55.993237 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.993206    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299"} err="failed to get container status \"4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299\": rpc error: code = NotFound desc = could not find container \"4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299\": container with ID starting with 4183445b8f8459c5980bb27eb92c7e91b56166597de3146b9313bf58c25c3299 not found: ID does not exist"
Sep 17 18:51:55.998980 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:55.998958    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"]
Sep 17 18:51:56.003232 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:56.003208    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1827-predictor-64748d747d-75nl8"]
Sep 17 18:51:56.852859 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:56.852823    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 18:51:57.063223 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:57.063191    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" path="/var/lib/kubelet/pods/12c747a1-b5ef-44d2-9cf1-dc75c416eab4/volumes"
Sep 17 18:51:59.976603 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:59.976574    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:51:59.976959 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:51:59.976934    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:52:06.853453 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:06.853426    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:52:09.977980 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:09.977945    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:52:19.977983 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:19.977940    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:52:28.442645 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.442607    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"]
Sep 17 18:52:28.443205 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.442898    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container" containerID="cri-o://6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5" gracePeriod=30
Sep 17 18:52:28.443205 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.442922    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kube-rbac-proxy" containerID="cri-o://68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2" gracePeriod=30
Sep 17 18:52:28.587593 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.587561    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"]
Sep 17 18:52:28.587850 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.587838    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kube-rbac-proxy"
Sep 17 18:52:28.587892 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.587852    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kube-rbac-proxy"
Sep 17 18:52:28.587892 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.587867    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container"
Sep 17 18:52:28.587892 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.587876    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container"
Sep 17 18:52:28.587986 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.587925    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kube-rbac-proxy"
Sep 17 18:52:28.587986 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.587933    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c747a1-b5ef-44d2-9cf1-dc75c416eab4" containerName="kserve-container"
Sep 17 18:52:28.590719 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.590705    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.593505 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.593483    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c4e90-kube-rbac-proxy-sar-config\""
Sep 17 18:52:28.593625 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.593482    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c4e90-predictor-serving-cert\""
Sep 17 18:52:28.601892 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.601873    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"]
Sep 17 18:52:28.701601 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.701520    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c4e90-predictor-7889b4769d-6wm5n\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") " pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.701601 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.701573    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhsqz\" (UniqueName: \"kubernetes.io/projected/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-api-access-lhsqz\") pod \"success-200-isvc-c4e90-predictor-7889b4769d-6wm5n\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") " pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.701601 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.701597    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-proxy-tls\") pod \"success-200-isvc-c4e90-predictor-7889b4769d-6wm5n\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") " pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.802086 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.802053    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c4e90-predictor-7889b4769d-6wm5n\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") " pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.802327 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.802104    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhsqz\" (UniqueName: \"kubernetes.io/projected/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-api-access-lhsqz\") pod \"success-200-isvc-c4e90-predictor-7889b4769d-6wm5n\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") " pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.802327 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.802126    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-proxy-tls\") pod \"success-200-isvc-c4e90-predictor-7889b4769d-6wm5n\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") " pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.802865 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.802837    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c4e90-predictor-7889b4769d-6wm5n\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") " pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.804663 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.804641    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-proxy-tls\") pod \"success-200-isvc-c4e90-predictor-7889b4769d-6wm5n\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") " pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.811704 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.811681    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhsqz\" (UniqueName: \"kubernetes.io/projected/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-api-access-lhsqz\") pod \"success-200-isvc-c4e90-predictor-7889b4769d-6wm5n\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") " pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:28.900822 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:28.900792    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:29.095591 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:29.095560    2552 generic.go:356] "Generic (PLEG): container finished" podID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerID="68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2" exitCode=2
Sep 17 18:52:29.095741 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:29.095604    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" event={"ID":"385b8e7a-1198-4d81-a8e4-992081cee5e7","Type":"ContainerDied","Data":"68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2"}
Sep 17 18:52:29.173196 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:29.173173    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"]
Sep 17 18:52:29.175102 ip-10-0-136-130 kubenswrapper[2552]: W0917 18:52:29.175074    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e9cf7d_eac2_4010_8ad0_f5ec3405e17f.slice/crio-6863dcd73002be38d73dba0084c967fa058992015ebcba0f78aa2b844ce50267 WatchSource:0}: Error finding container 6863dcd73002be38d73dba0084c967fa058992015ebcba0f78aa2b844ce50267: Status 404 returned error can't find the container with id 6863dcd73002be38d73dba0084c967fa058992015ebcba0f78aa2b844ce50267
Sep 17 18:52:29.977228 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:29.977196    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 18:52:30.100579 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:30.100541    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" event={"ID":"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f","Type":"ContainerStarted","Data":"b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702"}
Sep 17 18:52:30.100725 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:30.100603    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" event={"ID":"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f","Type":"ContainerStarted","Data":"49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575"}
Sep 17 18:52:30.100725 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:30.100625    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" event={"ID":"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f","Type":"ContainerStarted","Data":"6863dcd73002be38d73dba0084c967fa058992015ebcba0f78aa2b844ce50267"}
Sep 17 18:52:30.100822 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:30.100766    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:30.134026 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:30.133989    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" podStartSLOduration=2.13397731 podStartE2EDuration="2.13397731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:30.132561541 +0000 UTC m=+1391.342459705" watchObservedRunningTime="2026-09-17 18:52:30.13397731 +0000 UTC m=+1391.343875462"
Sep 17 18:52:31.103695 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.103661    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:31.104912 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.104888    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:52:31.587561 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.587539    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:52:31.723510 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.723437    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-rbac-proxy-sar-config\") pod \"385b8e7a-1198-4d81-a8e4-992081cee5e7\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") "
Sep 17 18:52:31.723510 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.723502    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b8e7a-1198-4d81-a8e4-992081cee5e7-proxy-tls\") pod \"385b8e7a-1198-4d81-a8e4-992081cee5e7\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") "
Sep 17 18:52:31.723679 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.723533    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fcfj\" (UniqueName: \"kubernetes.io/projected/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-api-access-9fcfj\") pod \"385b8e7a-1198-4d81-a8e4-992081cee5e7\" (UID: \"385b8e7a-1198-4d81-a8e4-992081cee5e7\") "
Sep 17 18:52:31.723790 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.723766    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "385b8e7a-1198-4d81-a8e4-992081cee5e7" (UID: "385b8e7a-1198-4d81-a8e4-992081cee5e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:31.725592 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.725572    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385b8e7a-1198-4d81-a8e4-992081cee5e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "385b8e7a-1198-4d81-a8e4-992081cee5e7" (UID: "385b8e7a-1198-4d81-a8e4-992081cee5e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:31.725694 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.725606    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-api-access-9fcfj" (OuterVolumeSpecName: "kube-api-access-9fcfj") pod "385b8e7a-1198-4d81-a8e4-992081cee5e7" (UID: "385b8e7a-1198-4d81-a8e4-992081cee5e7"). InnerVolumeSpecName "kube-api-access-9fcfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:52:31.824234 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.824200    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/385b8e7a-1198-4d81-a8e4-992081cee5e7-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:52:31.824234 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.824226    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fcfj\" (UniqueName: \"kubernetes.io/projected/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-api-access-9fcfj\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:52:31.824234 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:31.824237    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/385b8e7a-1198-4d81-a8e4-992081cee5e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 18:52:32.108715 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.108684    2552 generic.go:356] "Generic (PLEG): container finished" podID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerID="6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5" exitCode=0
Sep 17 18:52:32.109115 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.108737    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"
Sep 17 18:52:32.109115 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.108766    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" event={"ID":"385b8e7a-1198-4d81-a8e4-992081cee5e7","Type":"ContainerDied","Data":"6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5"}
Sep 17 18:52:32.109115 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.108832    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8" event={"ID":"385b8e7a-1198-4d81-a8e4-992081cee5e7","Type":"ContainerDied","Data":"98e4f5f94cfe87c3e1fa7f9fdd98dad32b342ec58fc0a0bb9d6d865f09c90fe2"}
Sep 17 18:52:32.109115 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.108858    2552 scope.go:117] "RemoveContainer" containerID="68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2"
Sep 17 18:52:32.109516 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.109488    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:52:32.116759 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.116743    2552 scope.go:117] "RemoveContainer" containerID="6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5"
Sep 17 18:52:32.123417 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.123401    2552 scope.go:117] "RemoveContainer" containerID="68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2"
Sep 17 18:52:32.123655 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:52:32.123636    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2\": container with ID starting with 68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2 not found: ID does not exist" containerID="68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2"
Sep 17 18:52:32.123732 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.123667    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2"} err="failed to get container status \"68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2\": rpc error: code = NotFound desc = could not find container \"68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2\": container with ID starting with 68932f27b62002257964422367ae51976bf25b32db0b9d2ce0a267382a5b2ae2 not found: ID does not exist"
Sep 17 18:52:32.123732 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.123689    2552 scope.go:117] "RemoveContainer" containerID="6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5"
Sep 17 18:52:32.123948 ip-10-0-136-130 kubenswrapper[2552]: E0917 18:52:32.123932    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5\": container with ID starting with 6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5 not found: ID does not exist" containerID="6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5"
Sep 17 18:52:32.123985 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.123953    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5"} err="failed to get container status \"6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5\": rpc error: code = NotFound desc = could not find container \"6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5\": container with ID starting with 6c2a5746809ccc52ba4986e9efbe5cf85af0da6a8b69ed45bb9143a12d8c91b5 not found: ID does not exist"
Sep 17 18:52:32.140922 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.140901    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"]
Sep 17 18:52:32.147395 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:32.147370    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-31957-predictor-bb49bf486-kkdf8"]
Sep 17 18:52:33.064996 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:33.064968    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" path="/var/lib/kubelet/pods/385b8e7a-1198-4d81-a8e4-992081cee5e7/volumes"
Sep 17 18:52:37.113679 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:37.113653    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:52:37.114118 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:37.114020    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:52:39.978142 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:39.978115    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 18:52:47.114899 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:47.114860    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:52:57.114142 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:52:57.114098    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:53:07.114954 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:53:07.114875    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:53:17.115050 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:53:17.115024    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 18:54:21.035909 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:54:21.035801    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:54:21.039134 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:54:21.038049    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:59:21.059410 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:59:21.059286    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 18:59:21.065262 ip-10-0-136-130 kubenswrapper[2552]: I0917 18:59:21.060105    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 19:01:07.046796 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.046765    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"]
Sep 17 19:01:07.047374 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.046997    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container" containerID="cri-o://55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340" gracePeriod=30
Sep 17 19:01:07.047374 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.047104    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kube-rbac-proxy" containerID="cri-o://732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062" gracePeriod=30
Sep 17 19:01:07.235710 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.235677    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"]
Sep 17 19:01:07.235959 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.235947    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container"
Sep 17 19:01:07.236012 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.235960    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container"
Sep 17 19:01:07.236012 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.235977    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kube-rbac-proxy"
Sep 17 19:01:07.236012 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.235983    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kube-rbac-proxy"
Sep 17 19:01:07.236111 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.236022    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kube-rbac-proxy"
Sep 17 19:01:07.236111 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.236032    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="385b8e7a-1198-4d81-a8e4-992081cee5e7" containerName="kserve-container"
Sep 17 19:01:07.238631 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.238616    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.241572 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.241544    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-77f5f-predictor-serving-cert\""
Sep 17 19:01:07.242147 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.242123    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-77f5f-kube-rbac-proxy-sar-config\""
Sep 17 19:01:07.250382 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.250360    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"]
Sep 17 19:01:07.388709 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.388675    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-proxy-tls\") pod \"success-200-isvc-77f5f-predictor-844f655df8-cpff7\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") " pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.388861 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.388727    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-77f5f-predictor-844f655df8-cpff7\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") " pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.388925 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.388853    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjhq\" (UniqueName: \"kubernetes.io/projected/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-api-access-jzjhq\") pod \"success-200-isvc-77f5f-predictor-844f655df8-cpff7\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") " pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.490093 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.490058    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjhq\" (UniqueName: \"kubernetes.io/projected/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-api-access-jzjhq\") pod \"success-200-isvc-77f5f-predictor-844f655df8-cpff7\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") " pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.490093 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.490097    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-proxy-tls\") pod \"success-200-isvc-77f5f-predictor-844f655df8-cpff7\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") " pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.490331 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.490123    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-77f5f-predictor-844f655df8-cpff7\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") " pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.490719 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.490700    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-77f5f-predictor-844f655df8-cpff7\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") " pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.492491 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.492471    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-proxy-tls\") pod \"success-200-isvc-77f5f-predictor-844f655df8-cpff7\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") " pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.500098 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.500078    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjhq\" (UniqueName: \"kubernetes.io/projected/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-api-access-jzjhq\") pod \"success-200-isvc-77f5f-predictor-844f655df8-cpff7\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") " pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.550660 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.550634    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:07.818775 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.818749    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"]
Sep 17 19:01:07.820703 ip-10-0-136-130 kubenswrapper[2552]: W0917 19:01:07.820675    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb726b5_a8a6_4ce0_8e59_82b2a28ed730.slice/crio-02f3bc949538a7346658852b483af6b58f905153a93d2d9fd8f7371af7ea69a7 WatchSource:0}: Error finding container 02f3bc949538a7346658852b483af6b58f905153a93d2d9fd8f7371af7ea69a7: Status 404 returned error can't find the container with id 02f3bc949538a7346658852b483af6b58f905153a93d2d9fd8f7371af7ea69a7
Sep 17 19:01:07.882048 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.882020    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" event={"ID":"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730","Type":"ContainerStarted","Data":"059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2"}
Sep 17 19:01:07.882155 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.882091    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" event={"ID":"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730","Type":"ContainerStarted","Data":"02f3bc949538a7346658852b483af6b58f905153a93d2d9fd8f7371af7ea69a7"}
Sep 17 19:01:07.883917 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.883833    2552 generic.go:356] "Generic (PLEG): container finished" podID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerID="732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062" exitCode=2
Sep 17 19:01:07.883917 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:07.883874    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" event={"ID":"eafa2187-a8f6-4883-9d2d-8731e1ba2952","Type":"ContainerDied","Data":"732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062"}
Sep 17 19:01:08.889339 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:08.889289    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" event={"ID":"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730","Type":"ContainerStarted","Data":"f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed"}
Sep 17 19:01:08.889739 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:08.889419    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:08.910085 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:08.910046    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" podStartSLOduration=1.9100332949999999 podStartE2EDuration="1.910033295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:08.908649353 +0000 UTC m=+1910.118547506" watchObservedRunningTime="2026-09-17 19:01:08.910033295 +0000 UTC m=+1910.119931447"
Sep 17 19:01:09.892598 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:09.892572    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:09.893698 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:09.893674    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:01:09.973463 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:09.973428    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 19:01:09.977977 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:09.977945    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:01:10.092987 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.092957    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 19:01:10.210765 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.210685    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-rbac-proxy-sar-config\") pod \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") "
Sep 17 19:01:10.210765 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.210736    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fjwp\" (UniqueName: \"kubernetes.io/projected/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-api-access-4fjwp\") pod \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") "
Sep 17 19:01:10.210765 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.210763    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafa2187-a8f6-4883-9d2d-8731e1ba2952-proxy-tls\") pod \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\" (UID: \"eafa2187-a8f6-4883-9d2d-8731e1ba2952\") "
Sep 17 19:01:10.211035 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.211013    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eafa2187-a8f6-4883-9d2d-8731e1ba2952" (UID: "eafa2187-a8f6-4883-9d2d-8731e1ba2952"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:10.212824 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.212789    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafa2187-a8f6-4883-9d2d-8731e1ba2952-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eafa2187-a8f6-4883-9d2d-8731e1ba2952" (UID: "eafa2187-a8f6-4883-9d2d-8731e1ba2952"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:10.212934 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.212823    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-api-access-4fjwp" (OuterVolumeSpecName: "kube-api-access-4fjwp") pod "eafa2187-a8f6-4883-9d2d-8731e1ba2952" (UID: "eafa2187-a8f6-4883-9d2d-8731e1ba2952"). InnerVolumeSpecName "kube-api-access-4fjwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:10.311773 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.311751    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:01:10.311883 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.311770    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fjwp\" (UniqueName: \"kubernetes.io/projected/eafa2187-a8f6-4883-9d2d-8731e1ba2952-kube-api-access-4fjwp\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:01:10.311883 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.311793    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eafa2187-a8f6-4883-9d2d-8731e1ba2952-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:01:10.897831 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.897796    2552 generic.go:356] "Generic (PLEG): container finished" podID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerID="55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340" exitCode=0
Sep 17 19:01:10.898297 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.897854    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"
Sep 17 19:01:10.898297 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.897879    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" event={"ID":"eafa2187-a8f6-4883-9d2d-8731e1ba2952","Type":"ContainerDied","Data":"55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340"}
Sep 17 19:01:10.898297 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.897934    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629" event={"ID":"eafa2187-a8f6-4883-9d2d-8731e1ba2952","Type":"ContainerDied","Data":"b4c178a07cddefdd99f7bc212050d996670a2c046663a158b65b49e333eb58b1"}
Sep 17 19:01:10.898297 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.897960    2552 scope.go:117] "RemoveContainer" containerID="732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062"
Sep 17 19:01:10.898517 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.898481    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:01:10.905963 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.905946    2552 scope.go:117] "RemoveContainer" containerID="55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340"
Sep 17 19:01:10.912973 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.912954    2552 scope.go:117] "RemoveContainer" containerID="732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062"
Sep 17 19:01:10.913214 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:01:10.913196    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062\": container with ID starting with 732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062 not found: ID does not exist" containerID="732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062"
Sep 17 19:01:10.913251 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.913222    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062"} err="failed to get container status \"732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062\": rpc error: code = NotFound desc = could not find container \"732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062\": container with ID starting with 732d89d96620f1f83bc79920e54abc20b55bad6ec19d5aee87d16439f74e6062 not found: ID does not exist"
Sep 17 19:01:10.913251 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.913237    2552 scope.go:117] "RemoveContainer" containerID="55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340"
Sep 17 19:01:10.913477 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:01:10.913457    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340\": container with ID starting with 55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340 not found: ID does not exist" containerID="55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340"
Sep 17 19:01:10.913525 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.913483    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340"} err="failed to get container status \"55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340\": rpc error: code = NotFound desc = could not find container \"55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340\": container with ID starting with 55a4c8f53c3bd8b3e64aa1ea975fd018de5e56aa84dafcde8b60515077d8c340 not found: ID does not exist"
Sep 17 19:01:10.921315 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.921284    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"]
Sep 17 19:01:10.924954 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:10.924934    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-10598-predictor-756678795d-kb629"]
Sep 17 19:01:11.064051 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:11.064009    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" path="/var/lib/kubelet/pods/eafa2187-a8f6-4883-9d2d-8731e1ba2952/volumes"
Sep 17 19:01:15.903181 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:15.903150    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:01:15.903787 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:15.903759    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:01:25.904355 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:25.904290    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:01:35.904409 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:35.904370    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:01:43.096908 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.096876    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"]
Sep 17 19:01:43.097376 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.097095    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container" containerID="cri-o://49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575" gracePeriod=30
Sep 17 19:01:43.097376 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.097201    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kube-rbac-proxy" containerID="cri-o://b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702" gracePeriod=30
Sep 17 19:01:43.232204 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.232174    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"]
Sep 17 19:01:43.232520 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.232502    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container"
Sep 17 19:01:43.232598 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.232521    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container"
Sep 17 19:01:43.232598 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.232559    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kube-rbac-proxy"
Sep 17 19:01:43.232598 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.232569    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kube-rbac-proxy"
Sep 17 19:01:43.232750 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.232639    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kube-rbac-proxy"
Sep 17 19:01:43.232750 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.232654    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafa2187-a8f6-4883-9d2d-8731e1ba2952" containerName="kserve-container"
Sep 17 19:01:43.235390 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.235370    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.238257 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.238238    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-29aad-predictor-serving-cert\""
Sep 17 19:01:43.238384 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.238263    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-29aad-kube-rbac-proxy-sar-config\""
Sep 17 19:01:43.244479 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.244449    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"]
Sep 17 19:01:43.338947 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.338916    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-962fc\" (UniqueName: \"kubernetes.io/projected/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-api-access-962fc\") pod \"success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") " pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.339088 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.338961    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-proxy-tls\") pod \"success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") " pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.339088 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.339019    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") " pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.440381 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.440279    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-962fc\" (UniqueName: \"kubernetes.io/projected/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-api-access-962fc\") pod \"success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") " pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.440527 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.440385    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-proxy-tls\") pod \"success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") " pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.440527 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.440435    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") " pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.441100 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.441079    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") " pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.442761 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.442738    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-proxy-tls\") pod \"success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") " pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.449165 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.449144    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-962fc\" (UniqueName: \"kubernetes.io/projected/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-api-access-962fc\") pod \"success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") " pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.546040 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.546008    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:43.816042 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:43.816017    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"]
Sep 17 19:01:43.818142 ip-10-0-136-130 kubenswrapper[2552]: W0917 19:01:43.818114    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade77fcd_71e5_4aa7_a5bc_4efe95b68c17.slice/crio-cabaf4fe4e625e7e6172477af7a907ca7fb2f4b4c488b4f56a4c99cbd5a6d73d WatchSource:0}: Error finding container cabaf4fe4e625e7e6172477af7a907ca7fb2f4b4c488b4f56a4c99cbd5a6d73d: Status 404 returned error can't find the container with id cabaf4fe4e625e7e6172477af7a907ca7fb2f4b4c488b4f56a4c99cbd5a6d73d
Sep 17 19:01:44.021129 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:44.021091    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" event={"ID":"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17","Type":"ContainerStarted","Data":"3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9"}
Sep 17 19:01:44.021286 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:44.021161    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:44.021394 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:44.021292    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:44.021394 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:44.021355    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" event={"ID":"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17","Type":"ContainerStarted","Data":"59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550"}
Sep 17 19:01:44.021394 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:44.021390    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" event={"ID":"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17","Type":"ContainerStarted","Data":"cabaf4fe4e625e7e6172477af7a907ca7fb2f4b4c488b4f56a4c99cbd5a6d73d"}
Sep 17 19:01:44.022825 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:44.022792    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 19:01:44.023158 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:44.023139    2552 generic.go:356] "Generic (PLEG): container finished" podID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerID="b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702" exitCode=2
Sep 17 19:01:44.023202 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:44.023181    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" event={"ID":"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f","Type":"ContainerDied","Data":"b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702"}
Sep 17 19:01:44.039342 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:44.039284    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podStartSLOduration=1.0392707350000001 podStartE2EDuration="1.039270735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:44.038149452 +0000 UTC m=+1945.248047596" watchObservedRunningTime="2026-09-17 19:01:44.039270735 +0000 UTC m=+1945.249168887"
Sep 17 19:01:45.026821 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:45.026778    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 19:01:45.904322 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:45.904262    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:01:46.235775 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.235750    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 19:01:46.260024 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.259992    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-rbac-proxy-sar-config\") pod \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") "
Sep 17 19:01:46.260186 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.260044    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhsqz\" (UniqueName: \"kubernetes.io/projected/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-api-access-lhsqz\") pod \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") "
Sep 17 19:01:46.260186 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.260073    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-proxy-tls\") pod \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\" (UID: \"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f\") "
Sep 17 19:01:46.260418 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.260386    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" (UID: "73e9cf7d-eac2-4010-8ad0-f5ec3405e17f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:46.262726 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.262696    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-api-access-lhsqz" (OuterVolumeSpecName: "kube-api-access-lhsqz") pod "73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" (UID: "73e9cf7d-eac2-4010-8ad0-f5ec3405e17f"). InnerVolumeSpecName "kube-api-access-lhsqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:46.262821 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.262786    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" (UID: "73e9cf7d-eac2-4010-8ad0-f5ec3405e17f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:46.361207 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.361175    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:01:46.361207 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.361203    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhsqz\" (UniqueName: \"kubernetes.io/projected/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-kube-api-access-lhsqz\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:01:46.361207 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:46.361214    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:01:47.037963 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.037928    2552 generic.go:356] "Generic (PLEG): container finished" podID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerID="49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575" exitCode=0
Sep 17 19:01:47.038207 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.037974    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" event={"ID":"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f","Type":"ContainerDied","Data":"49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575"}
Sep 17 19:01:47.038207 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.037980    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"
Sep 17 19:01:47.038207 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.038028    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n" event={"ID":"73e9cf7d-eac2-4010-8ad0-f5ec3405e17f","Type":"ContainerDied","Data":"6863dcd73002be38d73dba0084c967fa058992015ebcba0f78aa2b844ce50267"}
Sep 17 19:01:47.038207 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.038057    2552 scope.go:117] "RemoveContainer" containerID="b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702"
Sep 17 19:01:47.046330 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.046294    2552 scope.go:117] "RemoveContainer" containerID="49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575"
Sep 17 19:01:47.055341 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.055323    2552 scope.go:117] "RemoveContainer" containerID="b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702"
Sep 17 19:01:47.055897 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:01:47.055871    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702\": container with ID starting with b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702 not found: ID does not exist" containerID="b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702"
Sep 17 19:01:47.055984 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.055906    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702"} err="failed to get container status \"b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702\": rpc error: code = NotFound desc = could not find container \"b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702\": container with ID starting with b95810bf995e6847dacf34a58030a6ba6a1d90aa32b16c082dee0c40c71f2702 not found: ID does not exist"
Sep 17 19:01:47.055984 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.055926    2552 scope.go:117] "RemoveContainer" containerID="49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575"
Sep 17 19:01:47.056233 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:01:47.056211    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575\": container with ID starting with 49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575 not found: ID does not exist" containerID="49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575"
Sep 17 19:01:47.056291 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.056244    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575"} err="failed to get container status \"49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575\": rpc error: code = NotFound desc = could not find container \"49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575\": container with ID starting with 49e182da54f04bb139b3730fd04d7a9b36ed0dcf9abc30eb0c1b990c1c01b575 not found: ID does not exist"
Sep 17 19:01:47.066011 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.065990    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"]
Sep 17 19:01:47.068032 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:47.068011    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c4e90-predictor-7889b4769d-6wm5n"]
Sep 17 19:01:49.063913 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:49.063878    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" path="/var/lib/kubelet/pods/73e9cf7d-eac2-4010-8ad0-f5ec3405e17f/volumes"
Sep 17 19:01:50.032253 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:50.032226    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:01:50.032711 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:50.032686    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 19:01:55.905191 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:01:55.905165    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:02:00.033348 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:00.033277    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 19:02:10.033008 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:10.032923    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 19:02:17.426020 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.425990    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"]
Sep 17 19:02:17.426465 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.426221    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container" containerID="cri-o://059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2" gracePeriod=30
Sep 17 19:02:17.426465 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.426266    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kube-rbac-proxy" containerID="cri-o://f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed" gracePeriod=30
Sep 17 19:02:17.566948 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.566918    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"]
Sep 17 19:02:17.567214 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.567200    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container"
Sep 17 19:02:17.567256 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.567218    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container"
Sep 17 19:02:17.567256 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.567233    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kube-rbac-proxy"
Sep 17 19:02:17.567256 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.567242    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kube-rbac-proxy"
Sep 17 19:02:17.567385 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.567299    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kserve-container"
Sep 17 19:02:17.567385 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.567325    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e9cf7d-eac2-4010-8ad0-f5ec3405e17f" containerName="kube-rbac-proxy"
Sep 17 19:02:17.570040 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.570021    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:17.572783 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.572764    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5a2c4-kube-rbac-proxy-sar-config\""
Sep 17 19:02:17.572872 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.572856    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5a2c4-predictor-serving-cert\""
Sep 17 19:02:17.580677 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.580657    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"]
Sep 17 19:02:17.687955 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.687878    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d12821-8ccc-46cc-9075-9405d8b95630-proxy-tls\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:17.687955 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.687926    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d12821-8ccc-46cc-9075-9405d8b95630-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:17.687955 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.687947    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfdj\" (UniqueName: \"kubernetes.io/projected/56d12821-8ccc-46cc-9075-9405d8b95630-kube-api-access-rlfdj\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:17.788673 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.788636    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d12821-8ccc-46cc-9075-9405d8b95630-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:17.788673 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.788673    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfdj\" (UniqueName: \"kubernetes.io/projected/56d12821-8ccc-46cc-9075-9405d8b95630-kube-api-access-rlfdj\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:17.788902 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.788718    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d12821-8ccc-46cc-9075-9405d8b95630-proxy-tls\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:17.788902 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:02:17.788817    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-serving-cert: secret "success-200-isvc-5a2c4-predictor-serving-cert" not found
Sep 17 19:02:17.788902 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:02:17.788886    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56d12821-8ccc-46cc-9075-9405d8b95630-proxy-tls podName:56d12821-8ccc-46cc-9075-9405d8b95630 nodeName:}" failed. No retries permitted until 2026-09-17 19:02:18.288870511 +0000 UTC m=+1979.498768641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/56d12821-8ccc-46cc-9075-9405d8b95630-proxy-tls") pod "success-200-isvc-5a2c4-predictor-656b55656b-85ql9" (UID: "56d12821-8ccc-46cc-9075-9405d8b95630") : secret "success-200-isvc-5a2c4-predictor-serving-cert" not found
Sep 17 19:02:17.789324 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.789279    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d12821-8ccc-46cc-9075-9405d8b95630-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:17.801339 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:17.801288    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfdj\" (UniqueName: \"kubernetes.io/projected/56d12821-8ccc-46cc-9075-9405d8b95630-kube-api-access-rlfdj\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:18.152288 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:18.152251    2552 generic.go:356] "Generic (PLEG): container finished" podID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerID="f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed" exitCode=2
Sep 17 19:02:18.152531 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:18.152335    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" event={"ID":"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730","Type":"ContainerDied","Data":"f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed"}
Sep 17 19:02:18.292978 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:18.292944    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d12821-8ccc-46cc-9075-9405d8b95630-proxy-tls\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:18.295359 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:18.295328    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d12821-8ccc-46cc-9075-9405d8b95630-proxy-tls\") pod \"success-200-isvc-5a2c4-predictor-656b55656b-85ql9\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") " pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:18.480214 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:18.480144    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:18.763001 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:18.762466    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"]
Sep 17 19:02:18.766052 ip-10-0-136-130 kubenswrapper[2552]: W0917 19:02:18.766027    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d12821_8ccc_46cc_9075_9405d8b95630.slice/crio-ff366ae0fd0fd04658b6f074555409aa227de38a78e93e267cf981ae887db7c8 WatchSource:0}: Error finding container ff366ae0fd0fd04658b6f074555409aa227de38a78e93e267cf981ae887db7c8: Status 404 returned error can't find the container with id ff366ae0fd0fd04658b6f074555409aa227de38a78e93e267cf981ae887db7c8
Sep 17 19:02:19.158373 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:19.158343    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" event={"ID":"56d12821-8ccc-46cc-9075-9405d8b95630","Type":"ContainerStarted","Data":"74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2"}
Sep 17 19:02:19.158556 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:19.158419    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" event={"ID":"56d12821-8ccc-46cc-9075-9405d8b95630","Type":"ContainerStarted","Data":"fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90"}
Sep 17 19:02:19.158556 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:19.158466    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" event={"ID":"56d12821-8ccc-46cc-9075-9405d8b95630","Type":"ContainerStarted","Data":"ff366ae0fd0fd04658b6f074555409aa227de38a78e93e267cf981ae887db7c8"}
Sep 17 19:02:19.158556 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:19.158500    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:19.178056 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:19.178005    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" podStartSLOduration=2.177990912 podStartE2EDuration="2.177990912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:19.17687422 +0000 UTC m=+1980.386772372" watchObservedRunningTime="2026-09-17 19:02:19.177990912 +0000 UTC m=+1980.387889064"
Sep 17 19:02:20.033008 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.032968    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 19:02:20.163898 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.163862    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:20.164906 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.164871    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 19:02:20.777062 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.777039    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:02:20.914695 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.914665    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-proxy-tls\") pod \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") "
Sep 17 19:02:20.914853 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.914706    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-rbac-proxy-sar-config\") pod \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") "
Sep 17 19:02:20.914853 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.914843    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzjhq\" (UniqueName: \"kubernetes.io/projected/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-api-access-jzjhq\") pod \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\" (UID: \"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730\") "
Sep 17 19:02:20.915075 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.915053    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" (UID: "9fb726b5-a8a6-4ce0-8e59-82b2a28ed730"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:20.916780 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.916751    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" (UID: "9fb726b5-a8a6-4ce0-8e59-82b2a28ed730"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:20.916886 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:20.916812    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-api-access-jzjhq" (OuterVolumeSpecName: "kube-api-access-jzjhq") pod "9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" (UID: "9fb726b5-a8a6-4ce0-8e59-82b2a28ed730"). InnerVolumeSpecName "kube-api-access-jzjhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:21.016131 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.016104    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjhq\" (UniqueName: \"kubernetes.io/projected/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-api-access-jzjhq\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:02:21.016131 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.016126    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:02:21.016131 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.016135    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:02:21.168863 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.168798    2552 generic.go:356] "Generic (PLEG): container finished" podID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerID="059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2" exitCode=0
Sep 17 19:02:21.169174 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.168862    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"
Sep 17 19:02:21.169174 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.168887    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" event={"ID":"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730","Type":"ContainerDied","Data":"059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2"}
Sep 17 19:02:21.169174 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.168957    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7" event={"ID":"9fb726b5-a8a6-4ce0-8e59-82b2a28ed730","Type":"ContainerDied","Data":"02f3bc949538a7346658852b483af6b58f905153a93d2d9fd8f7371af7ea69a7"}
Sep 17 19:02:21.169174 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.168997    2552 scope.go:117] "RemoveContainer" containerID="f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed"
Sep 17 19:02:21.169553 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.169525    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 19:02:21.176492 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.176477    2552 scope.go:117] "RemoveContainer" containerID="059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2"
Sep 17 19:02:21.183124 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.183110    2552 scope.go:117] "RemoveContainer" containerID="f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed"
Sep 17 19:02:21.183514 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:02:21.183492    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed\": container with ID starting with f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed not found: ID does not exist" containerID="f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed"
Sep 17 19:02:21.183572 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.183521    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed"} err="failed to get container status \"f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed\": rpc error: code = NotFound desc = could not find container \"f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed\": container with ID starting with f8354b578b71b9e75d20383602b251869ebb3f845cfc78294ddc932a1b8483ed not found: ID does not exist"
Sep 17 19:02:21.183572 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.183536    2552 scope.go:117] "RemoveContainer" containerID="059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2"
Sep 17 19:02:21.183799 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:02:21.183780    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2\": container with ID starting with 059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2 not found: ID does not exist" containerID="059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2"
Sep 17 19:02:21.183844 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.183805    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2"} err="failed to get container status \"059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2\": rpc error: code = NotFound desc = could not find container \"059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2\": container with ID starting with 059b2e12865c6a69b25bd82e4327bffaca3eeda0b51ab54edc434f878fe366d2 not found: ID does not exist"
Sep 17 19:02:21.189983 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.189964    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"]
Sep 17 19:02:21.193490 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:21.193463    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f5f-predictor-844f655df8-cpff7"]
Sep 17 19:02:23.064390 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:23.064351    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" path="/var/lib/kubelet/pods/9fb726b5-a8a6-4ce0-8e59-82b2a28ed730/volumes"
Sep 17 19:02:26.173931 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:26.173899    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:02:26.174461 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:26.174432    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 19:02:30.033270 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:30.033242    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:02:36.174801 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:36.174767    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 19:02:46.174568 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:46.174527    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 19:02:56.174511 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:02:56.174470    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 19:03:06.175231 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:03:06.175194    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:04:21.081706 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:04:21.081607    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 19:04:21.084840 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:04:21.082340    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 19:09:21.102778 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:09:21.102755    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 19:09:21.105236 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:09:21.104289    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 19:11:32.341829 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:32.341756    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"]
Sep 17 19:11:32.342350 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:32.341982    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container" containerID="cri-o://fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90" gracePeriod=30
Sep 17 19:11:32.342350 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:32.342046    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kube-rbac-proxy" containerID="cri-o://74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2" gracePeriod=30
Sep 17 19:11:33.091932 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:33.091898    2552 generic.go:356] "Generic (PLEG): container finished" podID="56d12821-8ccc-46cc-9075-9405d8b95630" containerID="74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2" exitCode=2
Sep 17 19:11:33.092101 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:33.091972    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" event={"ID":"56d12821-8ccc-46cc-9075-9405d8b95630","Type":"ContainerDied","Data":"74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2"}
Sep 17 19:11:35.080485 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.080459    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:11:35.099266 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.099240    2552 generic.go:356] "Generic (PLEG): container finished" podID="56d12821-8ccc-46cc-9075-9405d8b95630" containerID="fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90" exitCode=0
Sep 17 19:11:35.099394 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.099281    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"
Sep 17 19:11:35.099394 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.099330    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" event={"ID":"56d12821-8ccc-46cc-9075-9405d8b95630","Type":"ContainerDied","Data":"fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90"}
Sep 17 19:11:35.099394 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.099392    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9" event={"ID":"56d12821-8ccc-46cc-9075-9405d8b95630","Type":"ContainerDied","Data":"ff366ae0fd0fd04658b6f074555409aa227de38a78e93e267cf981ae887db7c8"}
Sep 17 19:11:35.099552 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.099418    2552 scope.go:117] "RemoveContainer" containerID="74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2"
Sep 17 19:11:35.105627 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.105609    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlfdj\" (UniqueName: \"kubernetes.io/projected/56d12821-8ccc-46cc-9075-9405d8b95630-kube-api-access-rlfdj\") pod \"56d12821-8ccc-46cc-9075-9405d8b95630\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") "
Sep 17 19:11:35.105714 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.105657    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d12821-8ccc-46cc-9075-9405d8b95630-proxy-tls\") pod \"56d12821-8ccc-46cc-9075-9405d8b95630\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") "
Sep 17 19:11:35.105714 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.105693    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d12821-8ccc-46cc-9075-9405d8b95630-kube-rbac-proxy-sar-config\") pod \"56d12821-8ccc-46cc-9075-9405d8b95630\" (UID: \"56d12821-8ccc-46cc-9075-9405d8b95630\") "
Sep 17 19:11:35.106078 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.106055    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d12821-8ccc-46cc-9075-9405d8b95630-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56d12821-8ccc-46cc-9075-9405d8b95630" (UID: "56d12821-8ccc-46cc-9075-9405d8b95630"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:11:35.107538 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.107487    2552 scope.go:117] "RemoveContainer" containerID="fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90"
Sep 17 19:11:35.107903 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.107838    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d12821-8ccc-46cc-9075-9405d8b95630-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56d12821-8ccc-46cc-9075-9405d8b95630" (UID: "56d12821-8ccc-46cc-9075-9405d8b95630"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:11:35.108423 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.108380    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d12821-8ccc-46cc-9075-9405d8b95630-kube-api-access-rlfdj" (OuterVolumeSpecName: "kube-api-access-rlfdj") pod "56d12821-8ccc-46cc-9075-9405d8b95630" (UID: "56d12821-8ccc-46cc-9075-9405d8b95630"). InnerVolumeSpecName "kube-api-access-rlfdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:11:35.119885 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.119870    2552 scope.go:117] "RemoveContainer" containerID="74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2"
Sep 17 19:11:35.120203 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:11:35.120183    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2\": container with ID starting with 74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2 not found: ID does not exist" containerID="74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2"
Sep 17 19:11:35.120243 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.120211    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2"} err="failed to get container status \"74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2\": rpc error: code = NotFound desc = could not find container \"74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2\": container with ID starting with 74bcdf1b2b7681f2b5db2639f3d0331dc1a4e1cadf2f6de8ddc7e7eaaf7b34c2 not found: ID does not exist"
Sep 17 19:11:35.120243 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.120227    2552 scope.go:117] "RemoveContainer" containerID="fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90"
Sep 17 19:11:35.120488 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:11:35.120472    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90\": container with ID starting with fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90 not found: ID does not exist" containerID="fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90"
Sep 17 19:11:35.120528 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.120491    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90"} err="failed to get container status \"fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90\": rpc error: code = NotFound desc = could not find container \"fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90\": container with ID starting with fe22daa21170e05fb0db49df467bf07281791b5344e3324d01179757ebfa7e90 not found: ID does not exist"
Sep 17 19:11:35.206911 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.206885    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56d12821-8ccc-46cc-9075-9405d8b95630-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:11:35.206911 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.206908    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56d12821-8ccc-46cc-9075-9405d8b95630-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:11:35.207045 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.206919    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlfdj\" (UniqueName: \"kubernetes.io/projected/56d12821-8ccc-46cc-9075-9405d8b95630-kube-api-access-rlfdj\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:11:35.423115 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.423092    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"]
Sep 17 19:11:35.431262 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:35.431241    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a2c4-predictor-656b55656b-85ql9"]
Sep 17 19:11:37.063250 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:11:37.063212    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" path="/var/lib/kubelet/pods/56d12821-8ccc-46cc-9075-9405d8b95630/volumes"
Sep 17 19:14:21.124104 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:14:21.123999    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 19:14:21.128178 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:14:21.125538    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 19:19:02.620602 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:02.620569    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"]
Sep 17 19:19:02.621113 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:02.620779    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container" containerID="cri-o://59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550" gracePeriod=30
Sep 17 19:19:02.621113 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:02.620838    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kube-rbac-proxy" containerID="cri-o://3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9" gracePeriod=30
Sep 17 19:19:03.598502 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:03.598467    2552 generic.go:356] "Generic (PLEG): container finished" podID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerID="3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9" exitCode=2
Sep 17 19:19:03.598502 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:03.598496    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" event={"ID":"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17","Type":"ContainerDied","Data":"3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9"}
Sep 17 19:19:05.026988 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.026950    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 17 19:19:05.157933 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.157906    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:19:05.188247 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.188223    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-rbac-proxy-sar-config\") pod \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") "
Sep 17 19:19:05.188366 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.188269    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-962fc\" (UniqueName: \"kubernetes.io/projected/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-api-access-962fc\") pod \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") "
Sep 17 19:19:05.188366 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.188313    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-proxy-tls\") pod \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\" (UID: \"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17\") "
Sep 17 19:19:05.188549 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.188529    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" (UID: "ade77fcd-71e5-4aa7-a5bc-4efe95b68c17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:05.190280 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.190257    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" (UID: "ade77fcd-71e5-4aa7-a5bc-4efe95b68c17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:05.190456 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.190439    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-api-access-962fc" (OuterVolumeSpecName: "kube-api-access-962fc") pod "ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" (UID: "ade77fcd-71e5-4aa7-a5bc-4efe95b68c17"). InnerVolumeSpecName "kube-api-access-962fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:05.288867 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.288815    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-proxy-tls\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:19:05.288867 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.288837    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:19:05.288867 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.288847    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-962fc\" (UniqueName: \"kubernetes.io/projected/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17-kube-api-access-962fc\") on node \"ip-10-0-136-130.ec2.internal\" DevicePath \"\""
Sep 17 19:19:05.607179 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.607154    2552 generic.go:356] "Generic (PLEG): container finished" podID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerID="59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550" exitCode=0
Sep 17 19:19:05.607322 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.607199    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"
Sep 17 19:19:05.607322 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.607203    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" event={"ID":"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17","Type":"ContainerDied","Data":"59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550"}
Sep 17 19:19:05.607407 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.607326    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr" event={"ID":"ade77fcd-71e5-4aa7-a5bc-4efe95b68c17","Type":"ContainerDied","Data":"cabaf4fe4e625e7e6172477af7a907ca7fb2f4b4c488b4f56a4c99cbd5a6d73d"}
Sep 17 19:19:05.607407 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.607354    2552 scope.go:117] "RemoveContainer" containerID="3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9"
Sep 17 19:19:05.617655 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.617636    2552 scope.go:117] "RemoveContainer" containerID="59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550"
Sep 17 19:19:05.624361 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.624338    2552 scope.go:117] "RemoveContainer" containerID="3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9"
Sep 17 19:19:05.624645 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:19:05.624628    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9\": container with ID starting with 3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9 not found: ID does not exist" containerID="3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9"
Sep 17 19:19:05.624709 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.624653    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9"} err="failed to get container status \"3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9\": rpc error: code = NotFound desc = could not find container \"3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9\": container with ID starting with 3e282c193fd4a306ffb9932ed6e39f264ea6822a8f67c30ad2c4211ae9f258b9 not found: ID does not exist"
Sep 17 19:19:05.624709 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.624669    2552 scope.go:117] "RemoveContainer" containerID="59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550"
Sep 17 19:19:05.624918 ip-10-0-136-130 kubenswrapper[2552]: E0917 19:19:05.624902    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550\": container with ID starting with 59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550 not found: ID does not exist" containerID="59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550"
Sep 17 19:19:05.624955 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.624921    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550"} err="failed to get container status \"59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550\": rpc error: code = NotFound desc = could not find container \"59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550\": container with ID starting with 59e41e8f227ab59b89bf5a3525704fc8290da667829a5eb214f8b27caa778550 not found: ID does not exist"
Sep 17 19:19:05.629100 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.629078    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"]
Sep 17 19:19:05.636024 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:05.636001    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-29aad-predictor-6f6bfbbb58-gqwmr"]
Sep 17 19:19:07.063139 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:07.063108    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" path="/var/lib/kubelet/pods/ade77fcd-71e5-4aa7-a5bc-4efe95b68c17/volumes"
Sep 17 19:19:21.147645 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:21.147540    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 19:19:21.159513 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:21.150567    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tg7sn_bc894442-872d-4d68-9423-24d51bcc78fb/ovn-acl-logging/0.log"
Sep 17 19:19:34.118941 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.118905    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2jrbn/must-gather-87ztv"]
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119153    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119164    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119183    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kube-rbac-proxy"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119189    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kube-rbac-proxy"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119195    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kube-rbac-proxy"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119200    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kube-rbac-proxy"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119210    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119215    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119226    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119231    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119237    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kube-rbac-proxy"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119241    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kube-rbac-proxy"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119285    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kube-rbac-proxy"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119292    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kube-rbac-proxy"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119299    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kserve-container"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119319    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb726b5-a8a6-4ce0-8e59-82b2a28ed730" containerName="kserve-container"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119325    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade77fcd-71e5-4aa7-a5bc-4efe95b68c17" containerName="kserve-container"
Sep 17 19:19:34.119421 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.119334    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d12821-8ccc-46cc-9075-9405d8b95630" containerName="kube-rbac-proxy"
Sep 17 19:19:34.123313 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.123281    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jrbn/must-gather-87ztv"
Sep 17 19:19:34.126367 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.126348    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2jrbn\"/\"kube-root-ca.crt\""
Sep 17 19:19:34.126616 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.126602    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2jrbn\"/\"openshift-service-ca.crt\""
Sep 17 19:19:34.127677 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.127663    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2jrbn\"/\"default-dockercfg-4748h\""
Sep 17 19:19:34.133064 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.133042    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2jrbn/must-gather-87ztv"]
Sep 17 19:19:34.182733 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.182705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxzb\" (UniqueName: \"kubernetes.io/projected/e6acc1a3-98fa-4273-8a25-963e9012c54f-kube-api-access-fdxzb\") pod \"must-gather-87ztv\" (UID: \"e6acc1a3-98fa-4273-8a25-963e9012c54f\") " pod="openshift-must-gather-2jrbn/must-gather-87ztv"
Sep 17 19:19:34.182865 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.182741    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6acc1a3-98fa-4273-8a25-963e9012c54f-must-gather-output\") pod \"must-gather-87ztv\" (UID: \"e6acc1a3-98fa-4273-8a25-963e9012c54f\") " pod="openshift-must-gather-2jrbn/must-gather-87ztv"
Sep 17 19:19:34.284020 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.283986    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxzb\" (UniqueName: \"kubernetes.io/projected/e6acc1a3-98fa-4273-8a25-963e9012c54f-kube-api-access-fdxzb\") pod \"must-gather-87ztv\" (UID: \"e6acc1a3-98fa-4273-8a25-963e9012c54f\") " pod="openshift-must-gather-2jrbn/must-gather-87ztv"
Sep 17 19:19:34.284159 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.284033    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6acc1a3-98fa-4273-8a25-963e9012c54f-must-gather-output\") pod \"must-gather-87ztv\" (UID: \"e6acc1a3-98fa-4273-8a25-963e9012c54f\") " pod="openshift-must-gather-2jrbn/must-gather-87ztv"
Sep 17 19:19:34.284471 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.284452    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6acc1a3-98fa-4273-8a25-963e9012c54f-must-gather-output\") pod \"must-gather-87ztv\" (UID: \"e6acc1a3-98fa-4273-8a25-963e9012c54f\") " pod="openshift-must-gather-2jrbn/must-gather-87ztv"
Sep 17 19:19:34.294277 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.294259    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxzb\" (UniqueName: \"kubernetes.io/projected/e6acc1a3-98fa-4273-8a25-963e9012c54f-kube-api-access-fdxzb\") pod \"must-gather-87ztv\" (UID: \"e6acc1a3-98fa-4273-8a25-963e9012c54f\") " pod="openshift-must-gather-2jrbn/must-gather-87ztv"
Sep 17 19:19:34.432378 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.432320    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jrbn/must-gather-87ztv"
Sep 17 19:19:34.720211 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.720135    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2jrbn/must-gather-87ztv"]
Sep 17 19:19:34.722428 ip-10-0-136-130 kubenswrapper[2552]: W0917 19:19:34.722400    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6acc1a3_98fa_4273_8a25_963e9012c54f.slice/crio-5b3114d6ff21c04967aff66a6c38c999daa03bca35703cee58dd2ebb1f30938b WatchSource:0}: Error finding container 5b3114d6ff21c04967aff66a6c38c999daa03bca35703cee58dd2ebb1f30938b: Status 404 returned error can't find the container with id 5b3114d6ff21c04967aff66a6c38c999daa03bca35703cee58dd2ebb1f30938b
Sep 17 19:19:34.724528 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:34.724507    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:19:35.709609 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:35.709569    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jrbn/must-gather-87ztv" event={"ID":"e6acc1a3-98fa-4273-8a25-963e9012c54f","Type":"ContainerStarted","Data":"5b3114d6ff21c04967aff66a6c38c999daa03bca35703cee58dd2ebb1f30938b"}
Sep 17 19:19:36.716775 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:36.716737    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jrbn/must-gather-87ztv" event={"ID":"e6acc1a3-98fa-4273-8a25-963e9012c54f","Type":"ContainerStarted","Data":"9186bc3f0c1af3e83389ecfdbe99dd20f2ab653fb8b5e68d5ee8c44b79cb8ec9"}
Sep 17 19:19:36.717377 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:36.717351    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jrbn/must-gather-87ztv" event={"ID":"e6acc1a3-98fa-4273-8a25-963e9012c54f","Type":"ContainerStarted","Data":"e0e2168c962948f59809b3dbc0d612d8b9d67792201addb89995784db3d17672"}
Sep 17 19:19:36.737071 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:36.737010    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2jrbn/must-gather-87ztv" podStartSLOduration=1.704855392 podStartE2EDuration="2.736988944s" totalImagesPullingTime="1.032133552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:36.73591353 +0000 UTC m=+3017.945811683" watchObservedRunningTime="2026-09-17 19:19:36.736988944 +0000 UTC m=+3017.946887098"
Sep 17 19:19:37.139353 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:37.139325    2552 ???:1] "http: TLS handshake error from 10.0.143.176:38222: EOF"
Sep 17 19:19:37.148233 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:37.148208    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p45zn_e9337787-7c1b-4e8c-add0-59cd333bbd43/global-pull-secret-syncer/0.log"
Sep 17 19:19:37.443916 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:37.443843    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w2dj2_2afbcca0-e7f2-4fc9-badf-a2cbfe1a0c18/konnectivity-agent/0.log"
Sep 17 19:19:37.513951 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:37.513925    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-130.ec2.internal_7ad3ebf3ceb1ffa8eb4766b2c0e01b6b/haproxy/0.log"
Sep 17 19:19:42.877361 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:42.877327    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4z75_561e4771-4acc-42d1-86ae-51ab55a1384e/node-exporter/0.log"
Sep 17 19:19:42.943547 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:42.943466    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4z75_561e4771-4acc-42d1-86ae-51ab55a1384e/kube-rbac-proxy/0.log"
Sep 17 19:19:42.978131 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:42.978103    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4z75_561e4771-4acc-42d1-86ae-51ab55a1384e/init-textfile/0.log"
Sep 17 19:19:44.776719 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.776685    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"]
Sep 17 19:19:44.780179 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.780158    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.789429 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.789404    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"]
Sep 17 19:19:44.865928 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.865894    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-sys\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.866096 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.865948    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-podres\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.866096 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.866032    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-proc\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.866096 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.866068    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-lib-modules\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.866224 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.866114    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8plk\" (UniqueName: \"kubernetes.io/projected/61a18adf-6d9c-4938-bfc2-b6d39b768895-kube-api-access-c8plk\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.966987 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.966948    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8plk\" (UniqueName: \"kubernetes.io/projected/61a18adf-6d9c-4938-bfc2-b6d39b768895-kube-api-access-c8plk\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.967180 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.967017    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-sys\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.967180 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.967047    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-podres\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.967180 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.967093    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-proc\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.967180 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.967121    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-lib-modules\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.967424 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.967286    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-lib-modules\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.967567 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.967538    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-podres\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.967749 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.967686    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-proc\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.967749 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.967725    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61a18adf-6d9c-4938-bfc2-b6d39b768895-sys\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:44.975906 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:44.975881    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8plk\" (UniqueName: \"kubernetes.io/projected/61a18adf-6d9c-4938-bfc2-b6d39b768895-kube-api-access-c8plk\") pod \"perf-node-gather-daemonset-l9rgq\" (UID: \"61a18adf-6d9c-4938-bfc2-b6d39b768895\") " pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:45.093366 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:45.093298    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:45.386416 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:45.386390    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"]
Sep 17 19:19:45.388073 ip-10-0-136-130 kubenswrapper[2552]: W0917 19:19:45.388042    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod61a18adf_6d9c_4938_bfc2_b6d39b768895.slice/crio-ae0ae9cf59cead42afd5428863602e1075a521b9cf6887d33671c40a9767352d WatchSource:0}: Error finding container ae0ae9cf59cead42afd5428863602e1075a521b9cf6887d33671c40a9767352d: Status 404 returned error can't find the container with id ae0ae9cf59cead42afd5428863602e1075a521b9cf6887d33671c40a9767352d
Sep 17 19:19:45.760113 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:45.760041    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq" event={"ID":"61a18adf-6d9c-4938-bfc2-b6d39b768895","Type":"ContainerStarted","Data":"26058c2955a1363695daa12a900bf72686be9f3b2f81dabadfc39136095ccf78"}
Sep 17 19:19:45.760113 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:45.760093    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq" event={"ID":"61a18adf-6d9c-4938-bfc2-b6d39b768895","Type":"ContainerStarted","Data":"ae0ae9cf59cead42afd5428863602e1075a521b9cf6887d33671c40a9767352d"}
Sep 17 19:19:45.760296 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:45.760140    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:47.721703 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:47.721673    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rgs4g_a8f199ee-cba0-43b7-867f-8503480c9ff8/dns/0.log"
Sep 17 19:19:47.768487 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:47.768461    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rgs4g_a8f199ee-cba0-43b7-867f-8503480c9ff8/kube-rbac-proxy/0.log"
Sep 17 19:19:47.938717 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:47.938686    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-px26v_ba444217-b1b1-4f63-9579-d097b752acaa/dns-node-resolver/0.log"
Sep 17 19:19:48.623885 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:48.623855    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mgf76_9aa7c9d7-f8fb-4442-b34b-8df28ec8d9e7/node-ca/0.log"
Sep 17 19:19:49.968494 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:49.968462    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lfc2n_beecf21c-b666-4309-b7af-0165216fff83/serve-healthcheck-canary/0.log"
Sep 17 19:19:50.428491 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:50.428457    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4tjv2_17c175cf-5b8a-4201-bd88-88f28de96ec6/kube-rbac-proxy/0.log"
Sep 17 19:19:50.452508 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:50.452469    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4tjv2_17c175cf-5b8a-4201-bd88-88f28de96ec6/exporter/0.log"
Sep 17 19:19:50.487118 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:50.487091    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4tjv2_17c175cf-5b8a-4201-bd88-88f28de96ec6/extractor/0.log"
Sep 17 19:19:51.774129 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:51.774105    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq"
Sep 17 19:19:51.794763 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:51.794718    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2jrbn/perf-node-gather-daemonset-l9rgq" podStartSLOduration=7.794702008 podStartE2EDuration="7.794702008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:45.781570306 +0000 UTC m=+3026.991468467" watchObservedRunningTime="2026-09-17 19:19:51.794702008 +0000 UTC m=+3033.004600159"
Sep 17 19:19:53.243720 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:53.243660    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-dwbl9_1e61e9db-f164-4a2d-a90a-81f92f15a015/manager/0.log"
Sep 17 19:19:53.295562 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:53.295533    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-q7tsf_8d8cc8fb-2099-4e6e-9aac-312ee458c17b/s3-init/0.log"
Sep 17 19:19:53.313180 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:53.313157    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-q7tsf_8d8cc8fb-2099-4e6e-9aac-312ee458c17b/download-hf-model/0.log"
Sep 17 19:19:58.837996 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:58.837965    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-pppzq_260fbd36-e31c-48d5-bf5a-71f7b1d495a3/migrator/0.log"
Sep 17 19:19:58.882150 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:19:58.882124    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-pppzq_260fbd36-e31c-48d5-bf5a-71f7b1d495a3/graceful-termination/0.log"
Sep 17 19:20:00.667015 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:00.666977    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6swlj_ccbcf9b6-16ae-4b48-8661-a0d06e5d204b/kube-multus/0.log"
Sep 17 19:20:01.112806 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:01.112748    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9glr_ca1f1c1d-41b5-4d8f-99cd-925be255adf0/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:20:01.137295 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:01.137272    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9glr_ca1f1c1d-41b5-4d8f-99cd-925be255adf0/egress-router-binary-copy/0.log"
Sep 17 19:20:01.162994 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:01.162967    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9glr_ca1f1c1d-41b5-4d8f-99cd-925be255adf0/cni-plugins/0.log"
Sep 17 19:20:01.186409 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:01.186389    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9glr_ca1f1c1d-41b5-4d8f-99cd-925be255adf0/bond-cni-plugin/0.log"
Sep 17 19:20:01.210241 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:01.210217    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9glr_ca1f1c1d-41b5-4d8f-99cd-925be255adf0/routeoverride-cni/0.log"
Sep 17 19:20:01.240857 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:01.240829    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9glr_ca1f1c1d-41b5-4d8f-99cd-925be255adf0/whereabouts-cni-bincopy/0.log"
Sep 17 19:20:01.268045 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:01.267994    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9glr_ca1f1c1d-41b5-4d8f-99cd-925be255adf0/whereabouts-cni/0.log"
Sep 17 19:20:01.342662 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:01.342634    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bw2tp_876396f2-26fd-4a04-a6f0-00a1e8d41c3c/network-metrics-daemon/0.log"
Sep 17 19:20:01.377512 ip-10-0-136-130 kubenswrapper[2552]: I0917 19:20:01.377481    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bw2tp_876396f2-26fd-4a04-a6f0-00a1e8d41c3c/kube-rbac-proxy/0.log"