Sep 17 18:38:45.223560 ip-10-0-139-5 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:38:48.605176 ip-10-0-139-5 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:38:48.605176 ip-10-0-139-5 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:38:48.605176 ip-10-0-139-5 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:38:48.605176 ip-10-0-139-5 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:38:48.605176 ip-10-0-139-5 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:38:48.605997 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.605627    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:38:48.609716 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609693    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:48.609716 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609712    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:48.609716 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609716    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:48.609716 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609720    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:48.609716 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609723    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609726    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609729    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609732    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609735    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609738    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609740    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609743    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609746    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609748    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609751    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609754    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609757    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609759    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609764    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609768    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609771    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609774    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609777    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609780    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:48.609914 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609782    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609785    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609787    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609790    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609793    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609796    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609798    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609801    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609810    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609813    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609816    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609819    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609822    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609824    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609827    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609831    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609833    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609836    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609839    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:48.610428 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609843    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609847    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609850    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609853    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609855    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609858    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609861    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609863    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609866    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609869    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609871    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609874    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609877    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609879    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609882    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609884    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609887    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609890    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609892    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609895    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:48.610918 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609898    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609902    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609906    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609909    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609911    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609914    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609917    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609921    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609924    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609926    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609929    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609932    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609935    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609938    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609941    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609944    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609946    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609949    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609952    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:48.611419 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609954    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609958    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609961    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609964    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609966    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609969    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609972    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609974    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609977    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609981    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609983    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609987    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609990    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609993    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609996    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.609998    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610001    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610699    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610706    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:48.611866 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610709    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610712    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610715    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610718    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610721    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610725    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610729    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610733    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610736    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610739    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610742    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610745    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610748    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610751    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610753    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610756    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610760    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610762    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610765    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:48.612373 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610767    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610770    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610774    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610778    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610781    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610784    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610787    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610790    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610793    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610796    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610799    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610802    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610805    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610807    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610811    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610814    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610817    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610819    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610822    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610825    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:48.612874 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610827    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610830    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610833    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610835    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610837    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610840    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610842    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610845    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610849    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610853    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610856    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610859    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610862    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610864    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610868    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610870    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610873    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610876    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610879    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610881    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:48.613376 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610884    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610886    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610889    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610891    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610894    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610896    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610900    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610902    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610905    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610908    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610910    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610913    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610916    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610919    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610922    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610924    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610927    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610930    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610932    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:48.613861 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610935    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610937    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610941    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610949    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610952    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610955    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610957    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610960    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610964    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610967    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610969    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610972    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610974    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610977    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610979    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610982    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610985    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610988    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.610990    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611399    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:48.614338 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611404    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611409    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611412    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611416    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611419    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611423    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611426    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611429    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611431    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611434    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611437    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611439    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611442    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611445    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611447    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611450    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611456    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611458    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611461    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:48.614839 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611463    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611466    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611469    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611471    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611474    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611477    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611480    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611482    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611485    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611488    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611490    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611493    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611496    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611499    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611501    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611504    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611507    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611509    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611512    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611514    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:48.615305 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611517    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611520    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611522    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611525    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611527    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611530    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611534    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611537    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611541    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611545    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611549    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611553    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611556    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611558    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611561    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611563    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611566    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611568    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611571    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:48.615803 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611574    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611576    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611579    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611581    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611586    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611588    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611591    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611594    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611596    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611599    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611601    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611604    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611607    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611609    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611612    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611614    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611617    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611619    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611622    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611624    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:48.616271 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611627    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611629    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611633    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611638    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611640    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611644    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611647    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611649    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611652    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611655    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611657    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611660    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611662    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611665    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611667    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611670    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611674    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611677    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611679    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.611682    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:48.616769 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611757    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611798    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611806    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611811    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611815    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611818    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611823    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611827    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611830    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611833    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611837    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611841    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611844    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611846    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611851    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611854    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611858    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611861    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611865    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611868    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611872    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611875    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611878    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611882    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:38:48.617285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611885    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611888    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611892    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611895    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611899    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611902    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611905    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611908    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611912    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611915    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611917    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611920    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611923    2557 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611926    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611931    2557 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611934    2557 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611937    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611940    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611943    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611947    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611950    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611953    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611957    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611960    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611962    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:38:48.617864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611966    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611969    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611972    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611975    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611978    2557 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611982    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611984    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611988    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611991    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611994    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.611997    2557 flags.go:64] FLAG: --help="false"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612000    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-139-5.ec2.internal"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612004    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612007    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612010    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612013    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612017    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612020    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612023    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612026    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612028    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612031    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612034    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612038    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:38:48.618538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612041    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612043    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612046    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612049    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612052    2557 flags.go:64] FLAG: --lock-file=""
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612056    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612059    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612062    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612068    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612071    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612074    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612077    2557 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612079    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612083    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612086    2557 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612088    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612093    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612096    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612100    2557 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612103    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612106    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612109    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612112    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612115    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612118    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:38:48.619134 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612121    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612128    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612132    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612134    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612137    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612140    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612146    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612149    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612152    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612155    2557 flags.go:64] FLAG: --port="10250"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612158    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612161    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bc371d6fac85ef6d"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612164    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612168    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612172    2557 flags.go:64] FLAG: --register-node="true"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612175    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612179    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612182    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612185    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612188    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612191    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612195    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612198    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612201    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612204    2557 flags.go:64] FLAG: --runonce="false"
Sep 17 18:38:48.619770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612219    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612223    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612226    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612229    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612232    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612235    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612238    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612241    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612244    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612247    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612250    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612252    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612255    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612258    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612261    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612266    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612269    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612272    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612276    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612279    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612283    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612286    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612289    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612292    2557 flags.go:64] FLAG: --v="2"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612296    2557 flags.go:64] FLAG: --version="false"
Sep 17 18:38:48.620406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612300    2557 flags.go:64] FLAG: --vmodule=""
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612309    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.612319    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612437    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612441    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612443    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612446    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612449    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612453    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612456    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612459    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612461    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612464    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612467    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612471    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612474    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612477    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612479    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612482    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612484    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:48.621010 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612487    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612490    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612492    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612495    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612498    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612500    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612503    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612507    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612510    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612513    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612516    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612518    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612521    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612524    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612527    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612530    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612532    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612535    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612538    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612540    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:48.621528 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612544    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612547    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612549    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612552    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612554    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612557    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612559    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612562    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612565    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612567    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612570    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612572    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612575    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612577    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612580    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612582    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612585    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612589    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612592    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:48.622063 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612597    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612600    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612603    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612606    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612609    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612612    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612616    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612619    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612622    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612624    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612627    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612630    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612633    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612637    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612639    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612642    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612645    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612648    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612650    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612653    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:48.622559 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612655    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612658    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612661    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612663    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612666    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612669    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612671    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612674    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612677    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612679    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612682    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612685    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612689    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612692    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612694    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612698    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612702    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612705    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612709    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:48.623079 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612712    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612715    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612718    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.612721    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.613181    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.619998    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.620016    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.620036    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.620042    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620098    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620103    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620107    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620110    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620113    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:48.623563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620116    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620119    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620121    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620124    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620127    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620129    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620132    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620136    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620139    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620142    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620145    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620148    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620150    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620153    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620156    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620158    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620161    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620164    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620167    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:48.623930 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620170    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620173    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620176    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620178    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620182    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620185    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620187    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620190    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620192    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620195    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620198    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620200    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620203    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620205    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620225    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620228    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620232    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620236    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620239    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:48.624424 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620242    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620245    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620247    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620250    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620253    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620256    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620258    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620261    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620264    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620266    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620269    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620271    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620274    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620277    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620280    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620282    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620285    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620289    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620291    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620299    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:48.624894 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620302    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620304    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620307    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620315    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620321    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620324    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620328    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620330    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620333    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620336    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620339    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620342    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620345    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620348    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620350    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620353    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620355    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620358    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620361    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620363    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:48.625449 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620366    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620369    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620371    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620374    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620376    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620379    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620382    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620384    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620387    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620390    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620393    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620395    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620400    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620403    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620405    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:48.626014 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620408    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.620414    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620526    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620529    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620532    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620536    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620539    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620542    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620546    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620550    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620553    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620555    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620558    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620561    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:38:48.626563 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620563    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620566    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620569    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620571    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620573    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620576    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620578    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620581    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620583    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620588    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620591    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620595    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620598    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620600    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620603    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620606    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620609    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620612    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620614    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620617    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:38:48.626913 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620620    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620622    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620625    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620627    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620630    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620632    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620635    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620637    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620640    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620642    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620645    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620647    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620650    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620652    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620655    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620658    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620660    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620663    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620665    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620668    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:38:48.627417 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620671    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620673    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620676    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620678    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620681    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620684    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620686    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620688    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620691    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620694    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620697    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620699    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620702    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620704    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620707    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620710    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620712    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620715    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620717    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:38:48.628178 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620720    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620722    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620725    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620729    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620732    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620735    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620738    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620741    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620743    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620746    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620749    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620751    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620754    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620757    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620760    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620762    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620765    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620768    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620770    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620772    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:38:48.628911 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620775    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620778    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620781    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620783    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620786    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620789    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620791    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:48.620794    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.620799    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.620905    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.622819    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.623607    2557 server.go:1019] "Starting client certificate rotation"
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.623707    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:38:48.629491 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.623745    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:38:48.629828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.625874    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:38:48.629828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.627722    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:38:48.640947 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.640921    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:38:48.642498 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.642480    2557 log.go:25] "Validated CRI v1 image API"
Sep 17 18:38:48.643312 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.643298    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:38:48.646999 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.646981    2557 fs.go:135] Filesystem UUIDs: map[00d0aa15-afbc-47a7-a23a-60909bc15bc3:/dev/nvme0n1p3 53d22f00-9a82-4bfb-9478-f35bbcab239c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 18:38:48.647054 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.647000    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:38:48.651953 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.651848    2557 manager.go:216] Machine: {Timestamp:2026-09-17 18:38:48.651379102 +0000 UTC m=+2.006637886 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101163 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec242a90dda4851d6a1730b10a6e1fc9 SystemUUID:ec242a90-dda4-851d-6a17-30b10a6e1fc9 BootID:bdf0ec6a-44c2-4d4c-8afc-e243cafc401e Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:a2:46:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:a2:46:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:5f:5c:47:c3:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:38:48.651953 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.651951    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:38:48.652075 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.652063    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:38:48.653556 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.653530    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:38:48.653694 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.653560    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-5.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:38:48.654590 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.654579    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:38:48.654627 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.654592    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:38:48.654627 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.654600    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:38:48.654627 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.654611    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:38:48.654706 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.654682    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:38:48.657019 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.657002    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:38:48.658308 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.658297    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:38:48.658435 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.658426    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:38:48.658467 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.658462    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:38:48.658495 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.658482    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:38:48.658525 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.658507    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:38:48.658525 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.658521    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:38:48.658578 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.658533    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:38:48.658931 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.658919    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:38:48.658976 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.658939    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:38:48.660675 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.660660    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:38:48.663068 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.663054    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:38:48.663115 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.663086    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:38:48.664065 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664053    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:38:48.664100 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664071    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:38:48.664100 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664076    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:38:48.664100 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664081    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:38:48.664100 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664090    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:38:48.664100 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664095    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:38:48.664248 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664111    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:38:48.664248 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664119    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:38:48.664248 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664128    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:38:48.664248 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664134    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:38:48.664248 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664142    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:38:48.664248 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664147    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:38:48.664248 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664170    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:38:48.664248 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.664176    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:38:48.671367 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.670597    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:38:48.671888 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.671861    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:38:48.671965 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.671932    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:38:48.672012 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.672005    2557 server.go:1268] "Started kubelet"
Sep 17 18:38:48.672857 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.672822    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:38:48.672942 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.672839    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:38:48.672942 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.672916    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:38:48.672981 ip-10-0-139-5 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:38:48.674076 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.674037    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:38:48.675309 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.675292    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:38:48.677539 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.677515    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:38:48.680643 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.680632    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:38:48.680643 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.680636    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:38:48.680817 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.680779    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:48.681392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.681378    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:38:48.681529 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.681399    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:38:48.681584 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.681534    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:38:48.681659 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.681640    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:38:48.681659 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.681654    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:38:48.681659 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.681646    2557 factory.go:55] Registering systemd factory
Sep 17 18:38:48.681849 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.681680    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:38:48.681969 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.681953    2557 factory.go:153] Registering CRI-O factory
Sep 17 18:38:48.681969 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.681969    2557 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:38:48.682116 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.682037    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:38:48.682116 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.682056    2557 factory.go:103] Registering Raw factory
Sep 17 18:38:48.682116 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.682072    2557 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:38:48.682874 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.682854    2557 manager.go:349] Starting recovery of all containers
Sep 17 18:38:48.708446 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.708405    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:38:48.708624 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.708602    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:38:48.709247 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.708441    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-5.ec2.internal.18d62fae8d8c931a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-5.ec2.internal,UID:ip-10-0-139-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-5.ec2.internal,},FirstTimestamp:2026-09-17 18:38:48.671965978 +0000 UTC m=+2.027224746,LastTimestamp:2026-09-17 18:38:48.671965978 +0000 UTC m=+2.027224746,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-5.ec2.internal,}"
Sep 17 18:38:48.713011 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.712994    2557 manager.go:354] Recovery completed
Sep 17 18:38:48.715324 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.715305    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kjgds"
Sep 17 18:38:48.717071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.717058    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:48.718808 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.718787    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:48.718893 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.718883    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:48.718929 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.718898    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:48.719595 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.719581    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:38:48.719595 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.719593    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:38:48.719730 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.719608    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:38:48.719730 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.719649    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:48.720408 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.720393    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:48.720476 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.720423    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:48.720476 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.720437    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:48.720741 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.720675    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-5.ec2.internal.18d62fae9057544b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-5.ec2.internal,UID:ip-10-0-139-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-5.ec2.internal,},FirstTimestamp:2026-09-17 18:38:48.718808139 +0000 UTC m=+2.074066907,LastTimestamp:2026-09-17 18:38:48.718808139 +0000 UTC m=+2.074066907,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-5.ec2.internal,}"
Sep 17 18:38:48.721521 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.721506    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kjgds"
Sep 17 18:38:48.722849 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.722837    2557 policy_none.go:49] "None policy: Start"
Sep 17 18:38:48.722889 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.722852    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:38:48.722889 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.722863    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:38:48.724845 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.724834    2557 policy_none.go:47] "Start"
Sep 17 18:38:48.777133 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.776920    2557 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:38:48.777295 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.777162    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:38:48.777295 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.777177    2557 server.go:87] "Starting device plugin registration server"
Sep 17 18:38:48.777433 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.777422    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:38:48.777476 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.777442    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:38:48.777563 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.777532    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:38:48.777639 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.777626    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:38:48.777639 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.777635    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:38:48.778184 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.778168    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:38:48.778279 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.778244    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:48.828315 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.828265    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:38:48.829477 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.829458    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:38:48.829542 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.829526    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:38:48.829581 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.829554    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:38:48.829636 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.829619    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:38:48.831812 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.831790    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:48.877677 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.877590    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:48.878796 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.878779    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:48.878905 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.878826    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:48.878905 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.878841    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:48.878905 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.878869    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-5.ec2.internal"
Sep 17 18:38:48.890353 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.890331    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-5.ec2.internal"
Sep 17 18:38:48.890353 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.890358    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-5.ec2.internal\": node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:48.930369 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.929710    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal"]
Sep 17 18:38:48.930527 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.930406    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:48.932818 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.932794    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:48.932935 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.932826    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:48.932935 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.932836    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:48.934001 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.933982    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:48.935059 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.935046    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:48.935177 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.935158    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:48.935240 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.935182    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:48.936237 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.936207    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:48.936237 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.936231    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:48.936365 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.936249    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:48.936365 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.936262    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:48.936365 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.936251    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:48.936475 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.936379    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:48.938569 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.938554    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:48.938614 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.938587    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:38:48.939293 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.939279    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:38:48.939374 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.939310    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:38:48.939374 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.939324    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:38:48.970108 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.970077    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-5.ec2.internal\" not found" node="ip-10-0-139-5.ec2.internal"
Sep 17 18:38:48.973727 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:48.973706    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-5.ec2.internal\" not found" node="ip-10-0-139-5.ec2.internal"
Sep 17 18:38:48.983307 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.983275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27d844e4d22a0d2aa74300b1b468ed5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal\" (UID: \"27d844e4d22a0d2aa74300b1b468ed5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:48.983401 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.983309    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27d844e4d22a0d2aa74300b1b468ed5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal\" (UID: \"27d844e4d22a0d2aa74300b1b468ed5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:48.983401 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:48.983327    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0655c714803c176a16cc58f0c88d6bb7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-5.ec2.internal\" (UID: \"0655c714803c176a16cc58f0c88d6bb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.034521 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:49.034487    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:49.084384 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.084344    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27d844e4d22a0d2aa74300b1b468ed5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal\" (UID: \"27d844e4d22a0d2aa74300b1b468ed5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.084384 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.084387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27d844e4d22a0d2aa74300b1b468ed5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal\" (UID: \"27d844e4d22a0d2aa74300b1b468ed5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.084549 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.084403    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0655c714803c176a16cc58f0c88d6bb7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-5.ec2.internal\" (UID: \"0655c714803c176a16cc58f0c88d6bb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.084549 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.084446    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0655c714803c176a16cc58f0c88d6bb7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-5.ec2.internal\" (UID: \"0655c714803c176a16cc58f0c88d6bb7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.084549 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.084462    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27d844e4d22a0d2aa74300b1b468ed5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal\" (UID: \"27d844e4d22a0d2aa74300b1b468ed5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.084549 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.084463    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27d844e4d22a0d2aa74300b1b468ed5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal\" (UID: \"27d844e4d22a0d2aa74300b1b468ed5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.134649 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:49.134576    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:49.235298 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:49.235266    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:49.274449 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.274425    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.277594 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.277576    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.336002 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:49.335966    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:49.436586 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:49.436508    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:49.536986 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:49.536958    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:49.623527 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.623495    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:38:49.623950 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.623646    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:38:49.637783 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:49.637760    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:49.727074 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.727032    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:33:48 +0000 UTC" deadline="2028-06-18 06:49:35.830953614 +0000 UTC"
Sep 17 18:38:49.727074 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.727071    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15348h10m46.103887715s"
Sep 17 18:38:49.738221 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:49.738182    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-5.ec2.internal\" not found"
Sep 17 18:38:49.808220 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.808175    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:49.832110 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.832086    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:49.881548 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.881518    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.885880 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:49.885850    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0655c714803c176a16cc58f0c88d6bb7.slice/crio-dd50b408cb26192b8c6075f39190fa009478082ebcfa9bfb27f183af7e9b0a76 WatchSource:0}: Error finding container dd50b408cb26192b8c6075f39190fa009478082ebcfa9bfb27f183af7e9b0a76: Status 404 returned error can't find the container with id dd50b408cb26192b8c6075f39190fa009478082ebcfa9bfb27f183af7e9b0a76
Sep 17 18:38:49.886290 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:49.886268    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d844e4d22a0d2aa74300b1b468ed5e.slice/crio-cbb30ddfc45715978232441fab594536ee9a00713d510d7d486e73e9ff2c0842 WatchSource:0}: Error finding container cbb30ddfc45715978232441fab594536ee9a00713d510d7d486e73e9ff2c0842: Status 404 returned error can't find the container with id cbb30ddfc45715978232441fab594536ee9a00713d510d7d486e73e9ff2c0842
Sep 17 18:38:49.891666 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.891650    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:38:49.895981 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.895962    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:38:49.897474 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.897458    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal"
Sep 17 18:38:49.906803 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:49.906776    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:38:50.221312 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.221282    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:50.659864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.659829    2557 apiserver.go:52] "Watching apiserver"
Sep 17 18:38:50.671499 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.670909    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:38:50.671887 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.671853    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f","openshift-dns/node-resolver-8gdkz","openshift-image-registry/node-ca-4947k","openshift-multus/multus-additional-cni-plugins-wl4cz","openshift-multus/multus-lbftv","openshift-cluster-node-tuning-operator/tuned-jnmf8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal","openshift-multus/network-metrics-daemon-qvr8b","openshift-network-diagnostics/network-check-target-x7wf6","openshift-network-operator/iptables-alerter-7wll2","openshift-ovn-kubernetes/ovnkube-node-z6rd7","kube-system/global-pull-secret-syncer-thgrv","kube-system/konnectivity-agent-ns2kp"]
Sep 17 18:38:50.674837 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.674808    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.676972 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.676953    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.679440 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.679186    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.680235 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.680080    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:38:50.680852 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.680827    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:38:50.680973 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.680951    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v6bs9\""
Sep 17 18:38:50.681595 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.681164    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:38:50.681595 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.681286    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:38:50.681595 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.681396    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:38:50.681595 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.681556    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:38:50.682431 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.681657    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.682431 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.681695    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:38:50.682431 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.682001    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:38:50.682431 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.682040    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c9tth\""
Sep 17 18:38:50.683516 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.683361    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:38:50.683516 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.683394    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pmwvn\""
Sep 17 18:38:50.683516 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.683506    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:38:50.686225 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.686171    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.688830 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.688521    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.689433 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.689415    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:38:50.689526 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.689431    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:38:50.689707 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.689690    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jf7jl\""
Sep 17 18:38:50.690298 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.690278    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:38:50.691051 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.691034    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:50.691129 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.691098    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:38:50.692275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.691596    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-np227\""
Sep 17 18:38:50.692275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.691658    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:38:50.692275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.691850    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:38:50.692275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.692106    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:38:50.692275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.692187    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s9bdv\""
Sep 17 18:38:50.693081 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.692935    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:38:50.693488 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693371    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:50.693488 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693424    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-run-k8s-cni-cncf-io\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.693488 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693458    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-sysctl-d\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.693488 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.693430    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:38:50.693488 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693481    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-sys\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.693770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-var-lib-kubelet\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.693770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693559    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-system-cni-dir\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.693770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693584    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-socket-dir-parent\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.693770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-etc-kubernetes\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.693770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693640    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-modprobe-d\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.693770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693665    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-systemd\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.693770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693709    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-run\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.693770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693733    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-host\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.693770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a59040e-3d81-4961-8cd1-8fc477b7470e-tmp-dir\") pod \"node-resolver-8gdkz\" (UID: \"4a59040e-3d81-4961-8cd1-8fc477b7470e\") " pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693783    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-run-netns\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693806    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-var-lib-cni-bin\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-daemon-config\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-tuned\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693894    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9b7\" (UniqueName: \"kubernetes.io/projected/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-kube-api-access-4m9b7\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693916    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6vt\" (UniqueName: \"kubernetes.io/projected/4a59040e-3d81-4961-8cd1-8fc477b7470e-kube-api-access-zb6vt\") pod \"node-resolver-8gdkz\" (UID: \"4a59040e-3d81-4961-8cd1-8fc477b7470e\") " pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693939    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1b9b717-c3af-4187-b781-b6145b367908-host\") pod \"node-ca-4947k\" (UID: \"f1b9b717-c3af-4187-b781-b6145b367908\") " pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.693973    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-hostroot\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnnsq\" (UniqueName: \"kubernetes.io/projected/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-kube-api-access-bnnsq\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694043    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbd018c-8b15-4a81-a4b6-b96685c1afff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bbd018c-8b15-4a81-a4b6-b96685c1afff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694115    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqt9b\" (UniqueName: \"kubernetes.io/projected/9bbd018c-8b15-4a81-a4b6-b96685c1afff-kube-api-access-lqt9b\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694153    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-kubernetes\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.694183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1b9b717-c3af-4187-b781-b6145b367908-serviceca\") pod \"node-ca-4947k\" (UID: \"f1b9b717-c3af-4187-b781-b6145b367908\") " pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-cnibin\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694280    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-conf-dir\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694313    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-run-multus-certs\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694338    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bbd018c-8b15-4a81-a4b6-b96685c1afff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694378    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8vd\" (UniqueName: \"kubernetes.io/projected/f1b9b717-c3af-4187-b781-b6145b367908-kube-api-access-hb8vd\") pod \"node-ca-4947k\" (UID: \"f1b9b717-c3af-4187-b781-b6145b367908\") " pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694418    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-cni-dir\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694467    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-cni-binary-copy\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694499    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbd018c-8b15-4a81-a4b6-b96685c1afff-cnibin\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-var-lib-kubelet\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-sysconfig\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694583    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-sysctl-conf\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694615    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-tmp\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a59040e-3d81-4961-8cd1-8fc477b7470e-hosts-file\") pod \"node-resolver-8gdkz\" (UID: \"4a59040e-3d81-4961-8cd1-8fc477b7470e\") " pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-var-lib-cni-multus\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbd018c-8b15-4a81-a4b6-b96685c1afff-system-cni-dir\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbd018c-8b15-4a81-a4b6-b96685c1afff-cni-binary-copy\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.694958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.694738    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-lib-modules\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.696068 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.695701    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.698712 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.698691    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.701326 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.701311    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:50.701380 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.701366    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:38:50.701713 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.701511    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:38:50.701713 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.701557    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:38:50.703805 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.703634    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6vldp\""
Sep 17 18:38:50.706252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.704438    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:38:50.706252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.704455    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:38:50.706252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.704591    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:38:50.706252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.705155    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rrhvv\""
Sep 17 18:38:50.706252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.705527    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:38:50.706252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.705607    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:38:50.706252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.706027    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:38:50.706252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.706124    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:38:50.706252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.706206    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:38:50.708203 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.708185    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:38:50.709131 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.709001    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-txhbq\""
Sep 17 18:38:50.709131 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.709012    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:38:50.709409 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.709393    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:38:50.783555 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.783530    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:38:50.795898 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.795861    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-run-k8s-cni-cncf-io\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.796044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.795905    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-sys\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.796044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.795930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-var-lib-kubelet\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.796044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.795955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-cni-bin\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.796044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.795972    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6vt\" (UniqueName: \"kubernetes.io/projected/4a59040e-3d81-4961-8cd1-8fc477b7470e-kube-api-access-zb6vt\") pod \"node-resolver-8gdkz\" (UID: \"4a59040e-3d81-4961-8cd1-8fc477b7470e\") " pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.796044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.795985    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-host\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.796044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796024    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fb1140a-d7ef-4ac7-bb63-17106262554a-host-slash\") pod \"iptables-alerter-7wll2\" (UID: \"2fb1140a-d7ef-4ac7-bb63-17106262554a\") " pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.796044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttkl\" (UniqueName: \"kubernetes.io/projected/2fb1140a-d7ef-4ac7-bb63-17106262554a-kube-api-access-zttkl\") pod \"iptables-alerter-7wll2\" (UID: \"2fb1140a-d7ef-4ac7-bb63-17106262554a\") " pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796046    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-sys\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796124    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-host\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.795980    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-run-k8s-cni-cncf-io\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796187    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-var-lib-kubelet\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796235    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-cni-netd\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796254    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98765f76-dc15-47c1-ab00-28533582d24b-env-overrides\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796271    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc00a436-36cb-41da-b35c-c63466007e96-dbus\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796287    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-run-netns\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-daemon-config\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796341    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-tuned\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796382    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9b7\" (UniqueName: \"kubernetes.io/projected/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-kube-api-access-4m9b7\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.796412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-sys-fs\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796442    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-systemd-units\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796464    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-etc-openvswitch\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98765f76-dc15-47c1-ab00-28533582d24b-ovnkube-config\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796546    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1b9b717-c3af-4187-b781-b6145b367908-host\") pod \"node-ca-4947k\" (UID: \"f1b9b717-c3af-4187-b781-b6145b367908\") " pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796568    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-hostroot\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbd018c-8b15-4a81-a4b6-b96685c1afff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796636    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bbd018c-8b15-4a81-a4b6-b96685c1afff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796644    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f1b9b717-c3af-4187-b781-b6145b367908-host\") pod \"node-ca-4947k\" (UID: \"f1b9b717-c3af-4187-b781-b6145b367908\") " pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796671    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7l7w\" (UniqueName: \"kubernetes.io/projected/98765f76-dc15-47c1-ab00-28533582d24b-kube-api-access-b7l7w\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796691    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-hostroot\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796728    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1b9b717-c3af-4187-b781-b6145b367908-serviceca\") pod \"node-ca-4947k\" (UID: \"f1b9b717-c3af-4187-b781-b6145b367908\") " pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796780    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-cnibin\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796765    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796808    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-conf-dir\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796832    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-run-multus-certs\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796858    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:50.797025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796882    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8vd\" (UniqueName: \"kubernetes.io/projected/f1b9b717-c3af-4187-b781-b6145b367908-kube-api-access-hb8vd\") pod \"node-ca-4947k\" (UID: \"f1b9b717-c3af-4187-b781-b6145b367908\") " pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796904    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-cni-dir\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.796928    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbd018c-8b15-4a81-a4b6-b96685c1afff-cnibin\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797004    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-sysconfig\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797030    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a59040e-3d81-4961-8cd1-8fc477b7470e-hosts-file\") pod \"node-resolver-8gdkz\" (UID: \"4a59040e-3d81-4961-8cd1-8fc477b7470e\") " pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797044    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbd018c-8b15-4a81-a4b6-b96685c1afff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-socket-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbd018c-8b15-4a81-a4b6-b96685c1afff-system-cni-dir\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-lib-modules\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797174    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-conf-dir\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797195    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-run-multus-certs\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797238    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a59040e-3d81-4961-8cd1-8fc477b7470e-hosts-file\") pod \"node-resolver-8gdkz\" (UID: \"4a59040e-3d81-4961-8cd1-8fc477b7470e\") " pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797249    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbd018c-8b15-4a81-a4b6-b96685c1afff-system-cni-dir\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797278    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-sysconfig\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797281    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797321    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-cni-dir\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797310    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krs5x\" (UniqueName: \"kubernetes.io/projected/a0f20d3b-b7f6-435b-9375-bef4013e5de5-kube-api-access-krs5x\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:50.797834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-lib-modules\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797381    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbd018c-8b15-4a81-a4b6-b96685c1afff-cnibin\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-run-netns\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797519    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f1b9b717-c3af-4187-b781-b6145b367908-serviceca\") pod \"node-ca-4947k\" (UID: \"f1b9b717-c3af-4187-b781-b6145b367908\") " pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-cnibin\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-sysctl-d\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797612    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-daemon-config\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797633    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797674    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-device-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797692    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-sysctl-d\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797700    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfp5q\" (UniqueName: \"kubernetes.io/projected/0d990377-23e1-48f8-8574-bca87f71c957-kube-api-access-gfp5q\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797722    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-run-netns\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797776    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b22214b-faa3-490b-8e11-2ea7b4e6e15f-agent-certs\") pod \"konnectivity-agent-ns2kp\" (UID: \"3b22214b-faa3-490b-8e11-2ea7b4e6e15f\") " pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-system-cni-dir\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797835    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-socket-dir-parent\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797868    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-etc-kubernetes\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797891    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-modprobe-d\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.798702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-system-cni-dir\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797915    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-systemd\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797933    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-multus-socket-dir-parent\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797950    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-run\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797953    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-systemd\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797974    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a59040e-3d81-4961-8cd1-8fc477b7470e-tmp-dir\") pod \"node-resolver-8gdkz\" (UID: \"4a59040e-3d81-4961-8cd1-8fc477b7470e\") " pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797995    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-run\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.797993    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-etc-kubernetes\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98765f76-dc15-47c1-ab00-28533582d24b-ovn-node-metrics-cert\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-modprobe-d\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798008    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bbd018c-8b15-4a81-a4b6-b96685c1afff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798033    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-var-lib-cni-bin\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798064    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-var-lib-cni-bin\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-kubelet\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnnsq\" (UniqueName: \"kubernetes.io/projected/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-kube-api-access-bnnsq\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798157    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqt9b\" (UniqueName: \"kubernetes.io/projected/9bbd018c-8b15-4a81-a4b6-b96685c1afff-kube-api-access-lqt9b\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.799426 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798191    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-kubernetes\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798247    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bbd018c-8b15-4a81-a4b6-b96685c1afff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798253    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a59040e-3d81-4961-8cd1-8fc477b7470e-tmp-dir\") pod \"node-resolver-8gdkz\" (UID: \"4a59040e-3d81-4961-8cd1-8fc477b7470e\") " pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798271    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-run-systemd\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-run-ovn\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798325    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98765f76-dc15-47c1-ab00-28533582d24b-ovnkube-script-lib\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798358    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-cni-binary-copy\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-registration-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-kubernetes\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2fb1140a-d7ef-4ac7-bb63-17106262554a-iptables-alerter-script\") pod \"iptables-alerter-7wll2\" (UID: \"2fb1140a-d7ef-4ac7-bb63-17106262554a\") " pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798439    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-var-lib-openvswitch\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798464    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-log-socket\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798486    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc00a436-36cb-41da-b35c-c63466007e96-kubelet-config\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-var-lib-kubelet\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798559    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-sysctl-conf\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798644    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-var-lib-kubelet\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-sysctl-conf\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.800275 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798682    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-tmp\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798710    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bbd018c-8b15-4a81-a4b6-b96685c1afff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-slash\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798755    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b22214b-faa3-490b-8e11-2ea7b4e6e15f-konnectivity-ca\") pod \"konnectivity-agent-ns2kp\" (UID: \"3b22214b-faa3-490b-8e11-2ea7b4e6e15f\") " pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-var-lib-cni-multus\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798823    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbd018c-8b15-4a81-a4b6-b96685c1afff-cni-binary-copy\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-run-openvswitch\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798846    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-cni-binary-copy\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798825    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-host-var-lib-cni-multus\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798868    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-node-log\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.798906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.801071 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.799254    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbd018c-8b15-4a81-a4b6-b96685c1afff-cni-binary-copy\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.801618 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.801176    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-tmp\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.801618 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.801264    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-etc-tuned\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.809795 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.808935    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6vt\" (UniqueName: \"kubernetes.io/projected/4a59040e-3d81-4961-8cd1-8fc477b7470e-kube-api-access-zb6vt\") pod \"node-resolver-8gdkz\" (UID: \"4a59040e-3d81-4961-8cd1-8fc477b7470e\") " pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:50.809795 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.809069    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9b7\" (UniqueName: \"kubernetes.io/projected/8b2ec2ed-cfcb-4f35-8195-ccb69140aa71-kube-api-access-4m9b7\") pod \"tuned-jnmf8\" (UID: \"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71\") " pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:50.809795 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.809073    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8vd\" (UniqueName: \"kubernetes.io/projected/f1b9b717-c3af-4187-b781-b6145b367908-kube-api-access-hb8vd\") pod \"node-ca-4947k\" (UID: \"f1b9b717-c3af-4187-b781-b6145b367908\") " pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.810878 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.810855    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqt9b\" (UniqueName: \"kubernetes.io/projected/9bbd018c-8b15-4a81-a4b6-b96685c1afff-kube-api-access-lqt9b\") pod \"multus-additional-cni-plugins-wl4cz\" (UID: \"9bbd018c-8b15-4a81-a4b6-b96685c1afff\") " pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:50.812947 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.812927    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qjr6s"
Sep 17 18:38:50.814174 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.814153    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnnsq\" (UniqueName: \"kubernetes.io/projected/b92bcb7b-6ffd-4148-897b-eb21b6ed05df-kube-api-access-bnnsq\") pod \"multus-lbftv\" (UID: \"b92bcb7b-6ffd-4148-897b-eb21b6ed05df\") " pod="openshift-multus/multus-lbftv"
Sep 17 18:38:50.825927 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.825904    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qjr6s"
Sep 17 18:38:50.835033 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.834973    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal" event={"ID":"27d844e4d22a0d2aa74300b1b468ed5e","Type":"ContainerStarted","Data":"cbb30ddfc45715978232441fab594536ee9a00713d510d7d486e73e9ff2c0842"}
Sep 17 18:38:50.836336 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.836256    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal" event={"ID":"0655c714803c176a16cc58f0c88d6bb7","Type":"ContainerStarted","Data":"dd50b408cb26192b8c6075f39190fa009478082ebcfa9bfb27f183af7e9b0a76"}
Sep 17 18:38:50.900193 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-cni-netd\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900207    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98765f76-dc15-47c1-ab00-28533582d24b-env-overrides\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900248    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-cni-netd\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900246    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc00a436-36cb-41da-b35c-c63466007e96-dbus\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900301    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-sys-fs\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900332    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-systemd-units\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900337    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc00a436-36cb-41da-b35c-c63466007e96-dbus\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-etc-openvswitch\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900376    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-systemd-units\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98765f76-dc15-47c1-ab00-28533582d24b-ovnkube-config\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.900392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900384    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-sys-fs\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.900828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900408    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7l7w\" (UniqueName: \"kubernetes.io/projected/98765f76-dc15-47c1-ab00-28533582d24b-kube-api-access-b7l7w\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.900828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900414    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-etc-openvswitch\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.900828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900436    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:50.900828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900464    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:50.900828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900491    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-socket-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.900828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:50.900828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900540    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krs5x\" (UniqueName: \"kubernetes.io/projected/a0f20d3b-b7f6-435b-9375-bef4013e5de5-kube-api-access-krs5x\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:50.900828 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.900571    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900831    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.900870    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-device-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfp5q\" (UniqueName: \"kubernetes.io/projected/0d990377-23e1-48f8-8574-bca87f71c957-kube-api-access-gfp5q\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900940    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-run-netns\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.900958    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs podName:a0f20d3b-b7f6-435b-9375-bef4013e5de5 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:51.400936488 +0000 UTC m=+4.756195239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs") pod "network-metrics-daemon-qvr8b" (UID: "a0f20d3b-b7f6-435b-9375-bef4013e5de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.900989    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-run-netns\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901002    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b22214b-faa3-490b-8e11-2ea7b4e6e15f-agent-certs\") pod \"konnectivity-agent-ns2kp\" (UID: \"3b22214b-faa3-490b-8e11-2ea7b4e6e15f\") " pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901043    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98765f76-dc15-47c1-ab00-28533582d24b-ovn-node-metrics-cert\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901076    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-kubelet\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-run-systemd\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.901176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901160    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-run-ovn\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.901770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/98765f76-dc15-47c1-ab00-28533582d24b-ovnkube-config\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.901770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901318    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.901770 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.901451    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret podName:fc00a436-36cb-41da-b35c-c63466007e96 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:51.401421089 +0000 UTC m=+4.756679851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret") pod "global-pull-secret-syncer-thgrv" (UID: "fc00a436-36cb-41da-b35c-c63466007e96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:50.901770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901561    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-device-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.901770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901599    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/98765f76-dc15-47c1-ab00-28533582d24b-env-overrides\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.901770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901645    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-socket-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.901770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901718    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98765f76-dc15-47c1-ab00-28533582d24b-ovnkube-script-lib\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902086 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901841    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-kubelet\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902086 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901897    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-run-systemd\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902086 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.901948    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-run-ovn\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902235 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902119    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.902283 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902232    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/98765f76-dc15-47c1-ab00-28533582d24b-ovnkube-script-lib\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902327 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902280    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-registration-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.902379 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2fb1140a-d7ef-4ac7-bb63-17106262554a-iptables-alerter-script\") pod \"iptables-alerter-7wll2\" (UID: \"2fb1140a-d7ef-4ac7-bb63-17106262554a\") " pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.902379 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-var-lib-openvswitch\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902477 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-log-socket\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902477 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902402    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d990377-23e1-48f8-8574-bca87f71c957-registration-dir\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.902477 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902412    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc00a436-36cb-41da-b35c-c63466007e96-kubelet-config\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:50.902477 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902450    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-slash\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902664 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902480    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-var-lib-openvswitch\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902664 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902482    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b22214b-faa3-490b-8e11-2ea7b4e6e15f-konnectivity-ca\") pod \"konnectivity-agent-ns2kp\" (UID: \"3b22214b-faa3-490b-8e11-2ea7b4e6e15f\") " pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:38:50.902664 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902535    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-run-openvswitch\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902664 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902571    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-node-log\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902664 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902601    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902664 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902618    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc00a436-36cb-41da-b35c-c63466007e96-kubelet-config\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:50.902932 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902667    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-run-openvswitch\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902932 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-log-socket\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902932 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902830    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-node-log\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.902932 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902903    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2fb1140a-d7ef-4ac7-bb63-17106262554a-iptables-alerter-script\") pod \"iptables-alerter-7wll2\" (UID: \"2fb1140a-d7ef-4ac7-bb63-17106262554a\") " pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.903126 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.902977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-cni-bin\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.903126 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.903002    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-slash\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.903126 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.903059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fb1140a-d7ef-4ac7-bb63-17106262554a-host-slash\") pod \"iptables-alerter-7wll2\" (UID: \"2fb1140a-d7ef-4ac7-bb63-17106262554a\") " pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.903126 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.903090    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-cni-bin\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.903126 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.903113    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b22214b-faa3-490b-8e11-2ea7b4e6e15f-konnectivity-ca\") pod \"konnectivity-agent-ns2kp\" (UID: \"3b22214b-faa3-490b-8e11-2ea7b4e6e15f\") " pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:38:50.903379 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.903096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttkl\" (UniqueName: \"kubernetes.io/projected/2fb1140a-d7ef-4ac7-bb63-17106262554a-kube-api-access-zttkl\") pod \"iptables-alerter-7wll2\" (UID: \"2fb1140a-d7ef-4ac7-bb63-17106262554a\") " pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.903379 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.903145    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2fb1140a-d7ef-4ac7-bb63-17106262554a-host-slash\") pod \"iptables-alerter-7wll2\" (UID: \"2fb1140a-d7ef-4ac7-bb63-17106262554a\") " pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.903379 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.903161    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.903379 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.903166    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.903379 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.903249    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98765f76-dc15-47c1-ab00-28533582d24b-host-run-ovn-kubernetes\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.905112 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.904987    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/98765f76-dc15-47c1-ab00-28533582d24b-ovn-node-metrics-cert\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.905748 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.905634    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b22214b-faa3-490b-8e11-2ea7b4e6e15f-agent-certs\") pod \"konnectivity-agent-ns2kp\" (UID: \"3b22214b-faa3-490b-8e11-2ea7b4e6e15f\") " pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:38:50.911806 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.911736    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:50.911806 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.911761    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:50.911806 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.911775    2557 projected.go:196] Error preparing data for projected volume kube-api-access-jg4r9 for pod openshift-network-diagnostics/network-check-target-x7wf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:50.912012 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:50.911854    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9 podName:19f4f176-cbe7-4e1b-bb05-d7483eefe906 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:51.411837695 +0000 UTC m=+4.767096466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jg4r9" (UniqueName: "kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9") pod "network-check-target-x7wf6" (UID: "19f4f176-cbe7-4e1b-bb05-d7483eefe906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:50.913743 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.913716    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krs5x\" (UniqueName: \"kubernetes.io/projected/a0f20d3b-b7f6-435b-9375-bef4013e5de5-kube-api-access-krs5x\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:50.914088 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.914067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfp5q\" (UniqueName: \"kubernetes.io/projected/0d990377-23e1-48f8-8574-bca87f71c957-kube-api-access-gfp5q\") pod \"aws-ebs-csi-driver-node-bvc4f\" (UID: \"0d990377-23e1-48f8-8574-bca87f71c957\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:50.915050 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.915024    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7l7w\" (UniqueName: \"kubernetes.io/projected/98765f76-dc15-47c1-ab00-28533582d24b-kube-api-access-b7l7w\") pod \"ovnkube-node-z6rd7\" (UID: \"98765f76-dc15-47c1-ab00-28533582d24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:50.915937 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.915916    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttkl\" (UniqueName: \"kubernetes.io/projected/2fb1140a-d7ef-4ac7-bb63-17106262554a-kube-api-access-zttkl\") pod \"iptables-alerter-7wll2\" (UID: \"2fb1140a-d7ef-4ac7-bb63-17106262554a\") " pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:50.940656 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.940626    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:38:50.988681 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.988656    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4947k"
Sep 17 18:38:50.996143 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:50.996120    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lbftv"
Sep 17 18:38:51.007974 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.007955    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wl4cz"
Sep 17 18:38:51.015759 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.015734    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jnmf8"
Sep 17 18:38:51.021829 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.021804    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8gdkz"
Sep 17 18:38:51.028579 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.028554    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f"
Sep 17 18:38:51.035502 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.035480    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7wll2"
Sep 17 18:38:51.043219 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.043191    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:38:51.049930 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.049909    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:38:51.406545 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.406508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:51.406729 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.406577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:51.406729 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.406661    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:51.406729 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.406683    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:51.406878 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.406740    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret podName:fc00a436-36cb-41da-b35c-c63466007e96 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:52.40671781 +0000 UTC m=+5.761976584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret") pod "global-pull-secret-syncer-thgrv" (UID: "fc00a436-36cb-41da-b35c-c63466007e96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:51.406878 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.406764    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs podName:a0f20d3b-b7f6-435b-9375-bef4013e5de5 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:52.406752467 +0000 UTC m=+5.762011218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs") pod "network-metrics-daemon-qvr8b" (UID: "a0f20d3b-b7f6-435b-9375-bef4013e5de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:51.507070 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.507034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:51.507247 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.507226    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:51.507290 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.507253    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:51.507290 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.507265    2557 projected.go:196] Error preparing data for projected volume kube-api-access-jg4r9 for pod openshift-network-diagnostics/network-check-target-x7wf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:51.507362 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.507320    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9 podName:19f4f176-cbe7-4e1b-bb05-d7483eefe906 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:52.507304357 +0000 UTC m=+5.862563107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jg4r9" (UniqueName: "kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9") pod "network-check-target-x7wf6" (UID: "19f4f176-cbe7-4e1b-bb05-d7483eefe906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:51.732950 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:51.732914    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92bcb7b_6ffd_4148_897b_eb21b6ed05df.slice/crio-251f09e19af4d8c8175330631b66327506a3b60102251f4d1fd034e5c5038a41 WatchSource:0}: Error finding container 251f09e19af4d8c8175330631b66327506a3b60102251f4d1fd034e5c5038a41: Status 404 returned error can't find the container with id 251f09e19af4d8c8175330631b66327506a3b60102251f4d1fd034e5c5038a41
Sep 17 18:38:51.738399 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:51.738368    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2ec2ed_cfcb_4f35_8195_ccb69140aa71.slice/crio-2308a995c47d3bc924def89f0e3425d33e2dd9f4dc8c8af729bd28604625e951 WatchSource:0}: Error finding container 2308a995c47d3bc924def89f0e3425d33e2dd9f4dc8c8af729bd28604625e951: Status 404 returned error can't find the container with id 2308a995c47d3bc924def89f0e3425d33e2dd9f4dc8c8af729bd28604625e951
Sep 17 18:38:51.739961 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:51.739870    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b9b717_c3af_4187_b781_b6145b367908.slice/crio-775800efc8fa320c773c34e0557f046f167f2f3a4b85c072e6b5bbe26cd005e9 WatchSource:0}: Error finding container 775800efc8fa320c773c34e0557f046f167f2f3a4b85c072e6b5bbe26cd005e9: Status 404 returned error can't find the container with id 775800efc8fa320c773c34e0557f046f167f2f3a4b85c072e6b5bbe26cd005e9
Sep 17 18:38:51.740940 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:51.740622    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b22214b_faa3_490b_8e11_2ea7b4e6e15f.slice/crio-5e32982794b85e78966095916b8c83dea1ad4cf14bd81784b3c5f1e851253cf7 WatchSource:0}: Error finding container 5e32982794b85e78966095916b8c83dea1ad4cf14bd81784b3c5f1e851253cf7: Status 404 returned error can't find the container with id 5e32982794b85e78966095916b8c83dea1ad4cf14bd81784b3c5f1e851253cf7
Sep 17 18:38:51.741646 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:51.741621    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb1140a_d7ef_4ac7_bb63_17106262554a.slice/crio-2d51b32def909d9cc6817b4a8da17d2820622238f2941d11e3c018554dbc6f71 WatchSource:0}: Error finding container 2d51b32def909d9cc6817b4a8da17d2820622238f2941d11e3c018554dbc6f71: Status 404 returned error can't find the container with id 2d51b32def909d9cc6817b4a8da17d2820622238f2941d11e3c018554dbc6f71
Sep 17 18:38:51.743226 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:51.742781    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d990377_23e1_48f8_8574_bca87f71c957.slice/crio-0edd114337c459d3dabd42e22d674aeb83c3f2ca31741cbbb8306ad21de26078 WatchSource:0}: Error finding container 0edd114337c459d3dabd42e22d674aeb83c3f2ca31741cbbb8306ad21de26078: Status 404 returned error can't find the container with id 0edd114337c459d3dabd42e22d674aeb83c3f2ca31741cbbb8306ad21de26078
Sep 17 18:38:51.744062 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:51.744036    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a59040e_3d81_4961_8cd1_8fc477b7470e.slice/crio-1b566eaa3e572deba67a86d160bb13db4487334eef63d18dc18f1538dd54db41 WatchSource:0}: Error finding container 1b566eaa3e572deba67a86d160bb13db4487334eef63d18dc18f1538dd54db41: Status 404 returned error can't find the container with id 1b566eaa3e572deba67a86d160bb13db4487334eef63d18dc18f1538dd54db41
Sep 17 18:38:51.746260 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:51.745982    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbd018c_8b15_4a81_a4b6_b96685c1afff.slice/crio-6bb8d84158c4659ca5bf587db8a2204bf237845248d289b1eaa41be10bd29621 WatchSource:0}: Error finding container 6bb8d84158c4659ca5bf587db8a2204bf237845248d289b1eaa41be10bd29621: Status 404 returned error can't find the container with id 6bb8d84158c4659ca5bf587db8a2204bf237845248d289b1eaa41be10bd29621
Sep 17 18:38:51.747823 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:38:51.747748    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98765f76_dc15_47c1_ab00_28533582d24b.slice/crio-3cdf75bfc499b5252a4d21f25c776753dd1924e15b8b549d9cee6bc6d7beb64f WatchSource:0}: Error finding container 3cdf75bfc499b5252a4d21f25c776753dd1924e15b8b549d9cee6bc6d7beb64f: Status 404 returned error can't find the container with id 3cdf75bfc499b5252a4d21f25c776753dd1924e15b8b549d9cee6bc6d7beb64f
Sep 17 18:38:51.830291 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.830241    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:33:50 +0000 UTC" deadline="2028-03-26 15:20:09.816696462 +0000 UTC"
Sep 17 18:38:51.830291 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.830290    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13340h41m17.986410431s"
Sep 17 18:38:51.830471 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.830386    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:51.830471 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.830389    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:51.830539 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.830501    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:38:51.830617 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:51.830600    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:38:51.841974 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.841947    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerStarted","Data":"3cdf75bfc499b5252a4d21f25c776753dd1924e15b8b549d9cee6bc6d7beb64f"}
Sep 17 18:38:51.843154 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.843133    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wl4cz" event={"ID":"9bbd018c-8b15-4a81-a4b6-b96685c1afff","Type":"ContainerStarted","Data":"6bb8d84158c4659ca5bf587db8a2204bf237845248d289b1eaa41be10bd29621"}
Sep 17 18:38:51.844266 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.844244    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gdkz" event={"ID":"4a59040e-3d81-4961-8cd1-8fc477b7470e","Type":"ContainerStarted","Data":"1b566eaa3e572deba67a86d160bb13db4487334eef63d18dc18f1538dd54db41"}
Sep 17 18:38:51.847830 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.847803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7wll2" event={"ID":"2fb1140a-d7ef-4ac7-bb63-17106262554a","Type":"ContainerStarted","Data":"2d51b32def909d9cc6817b4a8da17d2820622238f2941d11e3c018554dbc6f71"}
Sep 17 18:38:51.850710 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.850687    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4947k" event={"ID":"f1b9b717-c3af-4187-b781-b6145b367908","Type":"ContainerStarted","Data":"775800efc8fa320c773c34e0557f046f167f2f3a4b85c072e6b5bbe26cd005e9"}
Sep 17 18:38:51.854199 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.854178    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f" event={"ID":"0d990377-23e1-48f8-8574-bca87f71c957","Type":"ContainerStarted","Data":"0edd114337c459d3dabd42e22d674aeb83c3f2ca31741cbbb8306ad21de26078"}
Sep 17 18:38:51.857696 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.857677    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ns2kp" event={"ID":"3b22214b-faa3-490b-8e11-2ea7b4e6e15f","Type":"ContainerStarted","Data":"5e32982794b85e78966095916b8c83dea1ad4cf14bd81784b3c5f1e851253cf7"}
Sep 17 18:38:51.859062 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.859042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jnmf8" event={"ID":"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71","Type":"ContainerStarted","Data":"2308a995c47d3bc924def89f0e3425d33e2dd9f4dc8c8af729bd28604625e951"}
Sep 17 18:38:51.860864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:51.860845    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lbftv" event={"ID":"b92bcb7b-6ffd-4148-897b-eb21b6ed05df","Type":"ContainerStarted","Data":"251f09e19af4d8c8175330631b66327506a3b60102251f4d1fd034e5c5038a41"}
Sep 17 18:38:52.413914 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.413633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:52.413914 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:52.413906    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:52.414142 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.413955    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:52.414142 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:52.413983    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret podName:fc00a436-36cb-41da-b35c-c63466007e96 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:54.413962259 +0000 UTC m=+7.769221029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret") pod "global-pull-secret-syncer-thgrv" (UID: "fc00a436-36cb-41da-b35c-c63466007e96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:52.414142 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:52.414038    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:52.414142 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:52.414089    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs podName:a0f20d3b-b7f6-435b-9375-bef4013e5de5 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:54.41407405 +0000 UTC m=+7.769332807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs") pod "network-metrics-daemon-qvr8b" (UID: "a0f20d3b-b7f6-435b-9375-bef4013e5de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:52.514384 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.514349    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:52.514565 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:52.514505    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:52.514565 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:52.514524    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:52.514565 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:52.514536    2557 projected.go:196] Error preparing data for projected volume kube-api-access-jg4r9 for pod openshift-network-diagnostics/network-check-target-x7wf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:52.514806 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:52.514591    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9 podName:19f4f176-cbe7-4e1b-bb05-d7483eefe906 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:54.514572445 +0000 UTC m=+7.869831219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jg4r9" (UniqueName: "kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9") pod "network-check-target-x7wf6" (UID: "19f4f176-cbe7-4e1b-bb05-d7483eefe906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:52.829859 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.829826    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:52.830469 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:52.829967    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:38:52.830469 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.830360    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:33:50 +0000 UTC" deadline="2028-03-07 20:57:50.88919539 +0000 UTC"
Sep 17 18:38:52.830469 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.830387    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12890h18m58.058813301s"
Sep 17 18:38:52.882709 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.882642    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal" event={"ID":"0655c714803c176a16cc58f0c88d6bb7","Type":"ContainerStarted","Data":"273172eed0c18658ecec54cc0fc9f62601e66100a84e1528e79a7b2e9260ffc5"}
Sep 17 18:38:52.887003 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.886978    2557 generic.go:356] "Generic (PLEG): container finished" podID="27d844e4d22a0d2aa74300b1b468ed5e" containerID="475f7c1f81a81760c84c8999f9ef668c35639385d3b6fe1eab9882ad9784da4e" exitCode=0
Sep 17 18:38:52.887118 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.887019    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal" event={"ID":"27d844e4d22a0d2aa74300b1b468ed5e","Type":"ContainerDied","Data":"475f7c1f81a81760c84c8999f9ef668c35639385d3b6fe1eab9882ad9784da4e"}
Sep 17 18:38:52.917847 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:52.917791    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-5.ec2.internal" podStartSLOduration=3.91776983 podStartE2EDuration="3.91776983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:52.901377925 +0000 UTC m=+6.256636699" watchObservedRunningTime="2026-09-17 18:38:52.91776983 +0000 UTC m=+6.273028586"
Sep 17 18:38:53.830601 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:53.830569    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:53.831039 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:53.830700    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:38:53.831100 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:53.831080    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:53.831192 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:53.831173    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:38:53.912930 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:53.912847    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal" event={"ID":"27d844e4d22a0d2aa74300b1b468ed5e","Type":"ContainerStarted","Data":"4607aba9e69cfcd124e65463880209ce84ab1b1ff72d73b3c5a4194030c46a78"}
Sep 17 18:38:53.934404 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:53.934348    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-5.ec2.internal" podStartSLOduration=4.934325711 podStartE2EDuration="4.934325711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:53.9334368 +0000 UTC m=+7.288695601" watchObservedRunningTime="2026-09-17 18:38:53.934325711 +0000 UTC m=+7.289584485"
Sep 17 18:38:54.431151 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:54.431114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:54.431395 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:54.431183    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:54.431395 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:54.431332    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:54.431511 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:54.431398    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs podName:a0f20d3b-b7f6-435b-9375-bef4013e5de5 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:58.431376468 +0000 UTC m=+11.786635240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs") pod "network-metrics-daemon-qvr8b" (UID: "a0f20d3b-b7f6-435b-9375-bef4013e5de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:54.431884 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:54.431781    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:54.431884 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:54.431838    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret podName:fc00a436-36cb-41da-b35c-c63466007e96 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:58.431821589 +0000 UTC m=+11.787080345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret") pod "global-pull-secret-syncer-thgrv" (UID: "fc00a436-36cb-41da-b35c-c63466007e96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:54.532529 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:54.532478    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:54.532713 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:54.532668    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:54.532713 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:54.532691    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:54.532713 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:54.532705    2557 projected.go:196] Error preparing data for projected volume kube-api-access-jg4r9 for pod openshift-network-diagnostics/network-check-target-x7wf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:54.532874 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:54.532762    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9 podName:19f4f176-cbe7-4e1b-bb05-d7483eefe906 nodeName:}" failed. No retries permitted until 2026-09-17 18:38:58.53274486 +0000 UTC m=+11.888003625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jg4r9" (UniqueName: "kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9") pod "network-check-target-x7wf6" (UID: "19f4f176-cbe7-4e1b-bb05-d7483eefe906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:54.832410 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:54.831805    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:54.832410 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:54.831966    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:38:55.831176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:55.830574    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:55.831176 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:55.830706    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:38:55.831176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:55.830775    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:55.831176 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:55.830859    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:38:56.829872 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:56.829836    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:56.830315 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:56.829973    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:38:57.830366 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:57.830326    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:57.830852 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:57.830458    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:38:57.830852 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:57.830830    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:57.831009 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:57.830973    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:38:58.466697 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:58.466657    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:58.466876 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:58.466834    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:58.466932 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:58.466910    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret podName:fc00a436-36cb-41da-b35c-c63466007e96 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:06.466888864 +0000 UTC m=+19.822147620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret") pod "global-pull-secret-syncer-thgrv" (UID: "fc00a436-36cb-41da-b35c-c63466007e96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:38:58.467206 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:58.467170    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:58.467414 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:58.467368    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:58.467526 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:58.467423    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs podName:a0f20d3b-b7f6-435b-9375-bef4013e5de5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:06.467407277 +0000 UTC m=+19.822666037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs") pod "network-metrics-daemon-qvr8b" (UID: "a0f20d3b-b7f6-435b-9375-bef4013e5de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:38:58.568570 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:58.568533    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:58.568709 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:58.568667    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:38:58.568709 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:58.568685    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:38:58.568709 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:58.568695    2557 projected.go:196] Error preparing data for projected volume kube-api-access-jg4r9 for pod openshift-network-diagnostics/network-check-target-x7wf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:58.568817 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:58.568742    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9 podName:19f4f176-cbe7-4e1b-bb05-d7483eefe906 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:06.568728364 +0000 UTC m=+19.923987116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jg4r9" (UniqueName: "kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9") pod "network-check-target-x7wf6" (UID: "19f4f176-cbe7-4e1b-bb05-d7483eefe906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:38:58.831025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:58.830962    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:38:58.831486 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:58.831074    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:38:59.830349 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:59.829834    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:38:59.830349 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:38:59.829876    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:38:59.830349 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:59.829966    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:38:59.830349 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:38:59.830075    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:00.830449 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:00.830417    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:00.830853 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:00.830528    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:01.829798 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:01.829762    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:01.829798 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:01.829796    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:01.830027 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:01.829894    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:01.830085 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:01.830020    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:02.829964 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:02.829927    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:02.830535 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:02.830065    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:03.829936 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:03.829900    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:03.830115 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:03.829939    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:03.830115 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:03.830040    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:03.830483 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:03.830183    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:04.830540 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:04.830503    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:04.831001 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:04.830635    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:05.830065 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:05.830026    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:05.830065 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:05.830071    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:05.830358 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:05.830164    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:05.830358 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:05.830319    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:06.528371 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:06.528315    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:06.528826 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:06.528391    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:06.528826 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:06.528513    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:06.528826 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:06.528598    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs podName:a0f20d3b-b7f6-435b-9375-bef4013e5de5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:22.528576898 +0000 UTC m=+35.883835656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs") pod "network-metrics-daemon-qvr8b" (UID: "a0f20d3b-b7f6-435b-9375-bef4013e5de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:06.528826 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:06.528513    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:06.528826 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:06.528661    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret podName:fc00a436-36cb-41da-b35c-c63466007e96 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:22.528644177 +0000 UTC m=+35.883902942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret") pod "global-pull-secret-syncer-thgrv" (UID: "fc00a436-36cb-41da-b35c-c63466007e96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:06.629515 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:06.629477    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:06.629699 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:06.629673    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:06.629771 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:06.629706    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:06.629771 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:06.629721    2557 projected.go:196] Error preparing data for projected volume kube-api-access-jg4r9 for pod openshift-network-diagnostics/network-check-target-x7wf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:06.629853 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:06.629790    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9 podName:19f4f176-cbe7-4e1b-bb05-d7483eefe906 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:22.629772269 +0000 UTC m=+35.985031036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jg4r9" (UniqueName: "kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9") pod "network-check-target-x7wf6" (UID: "19f4f176-cbe7-4e1b-bb05-d7483eefe906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:06.830763 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:06.830724    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:06.830923 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:06.830861    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:07.830777 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:07.830740    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:07.831242 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:07.830744    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:07.831242 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:07.830862    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:07.831242 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:07.830941    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:08.830715 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:08.830680    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:08.830897 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:08.830763    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:09.830673 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.830517    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:09.830754 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.830581    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:09.830986 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:09.830759    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:09.830986 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:09.830847    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:09.943487 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.943453    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lbftv" event={"ID":"b92bcb7b-6ffd-4148-897b-eb21b6ed05df","Type":"ContainerStarted","Data":"ee8b888f9462f4f6357592def48bac9bcf6eda96150afb2e112c08ef00239f14"}
Sep 17 18:39:09.946335 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.946314    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:39:09.946717 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.946695    2557 generic.go:356] "Generic (PLEG): container finished" podID="98765f76-dc15-47c1-ab00-28533582d24b" containerID="8438db538faabf428ecf304428580fc4cb4794bf2aa73ce523b999744f453aaf" exitCode=1
Sep 17 18:39:09.946824 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.946763    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerStarted","Data":"30919efd6bea84de5edea760026c5d0ed8f1be6416f572cb564b1ebf9ed66e02"}
Sep 17 18:39:09.946824 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.946811    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerStarted","Data":"c0cc1e9b0fe7d0aef798281d2bb0b16d0a40aaf67388555bb2c6af7e94eb0924"}
Sep 17 18:39:09.946934 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.946846    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerStarted","Data":"74cddb23850a5d633ce35077bc887896fad1ca3e54cf5cd3d1260d92f9cd9392"}
Sep 17 18:39:09.946934 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.946881    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerDied","Data":"8438db538faabf428ecf304428580fc4cb4794bf2aa73ce523b999744f453aaf"}
Sep 17 18:39:09.946934 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.946912    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerStarted","Data":"1bccf3b7527e343e69e8dc8de8d759d07478353eef6d52e5e204521b6eb80e5f"}
Sep 17 18:39:09.948770 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.948746    2557 generic.go:356] "Generic (PLEG): container finished" podID="9bbd018c-8b15-4a81-a4b6-b96685c1afff" containerID="655b2a5dea7d031b6b0b2f981a45634a5ee22e429cb048bf8b442b4ead8c3ec9" exitCode=0
Sep 17 18:39:09.948858 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.948774    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wl4cz" event={"ID":"9bbd018c-8b15-4a81-a4b6-b96685c1afff","Type":"ContainerDied","Data":"655b2a5dea7d031b6b0b2f981a45634a5ee22e429cb048bf8b442b4ead8c3ec9"}
Sep 17 18:39:09.950583 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.950526    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8gdkz" event={"ID":"4a59040e-3d81-4961-8cd1-8fc477b7470e","Type":"ContainerStarted","Data":"c25e92e172257000fffdc40dc28bf9c4772d4bdb887a9092bc5a7ee33582a8e4"}
Sep 17 18:39:09.953406 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.953138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4947k" event={"ID":"f1b9b717-c3af-4187-b781-b6145b367908","Type":"ContainerStarted","Data":"c0f747b08d64b18d05271feea0300a8a7ed579293a36371d4a30c493e3f96a82"}
Sep 17 18:39:09.957261 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.957169    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f" event={"ID":"0d990377-23e1-48f8-8574-bca87f71c957","Type":"ContainerStarted","Data":"aa1cb0cd3c2f168dd235ed6b85003dbab3b77e87925d3299c22c067bb1a67c4e"}
Sep 17 18:39:09.959110 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.959087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ns2kp" event={"ID":"3b22214b-faa3-490b-8e11-2ea7b4e6e15f","Type":"ContainerStarted","Data":"f3003df70c2b88f5880d3991f50ed356c8bcf116d7007f1749afb929dddd383a"}
Sep 17 18:39:09.960780 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.960758    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jnmf8" event={"ID":"8b2ec2ed-cfcb-4f35-8195-ccb69140aa71","Type":"ContainerStarted","Data":"7538fc3080857af9172ca39fde8f8308c67d51ff7159d960403bd21301da36c4"}
Sep 17 18:39:09.969176 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.969129    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lbftv" podStartSLOduration=4.496035589 podStartE2EDuration="21.969114143s" totalImagesPullingTime="17.473078554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:09.967247223 +0000 UTC m=+23.322506016" watchObservedRunningTime="2026-09-17 18:39:09.969114143 +0000 UTC m=+23.324372916"
Sep 17 18:39:09.994774 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:09.994722    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8gdkz" podStartSLOduration=3.544600278 podStartE2EDuration="20.994702683s" totalImagesPullingTime="17.450102405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:09.994594315 +0000 UTC m=+23.349853115" watchObservedRunningTime="2026-09-17 18:39:09.994702683 +0000 UTC m=+23.349961459"
Sep 17 18:39:10.028750 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.028692    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ns2kp" podStartSLOduration=3.627338807 podStartE2EDuration="21.02867147s" totalImagesPullingTime="17.401332663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:10.011624085 +0000 UTC m=+23.366882857" watchObservedRunningTime="2026-09-17 18:39:10.02867147 +0000 UTC m=+23.383930244"
Sep 17 18:39:10.029003 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.028968    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4947k" podStartSLOduration=4.604353639 podStartE2EDuration="22.028960665s" totalImagesPullingTime="17.424607026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:10.028339994 +0000 UTC m=+23.383598785" watchObservedRunningTime="2026-09-17 18:39:10.028960665 +0000 UTC m=+23.384219440"
Sep 17 18:39:10.816420 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.816178    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:39:10.830600 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.830577    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:10.830706 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:10.830677    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:10.964779 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.964690    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7wll2" event={"ID":"2fb1140a-d7ef-4ac7-bb63-17106262554a","Type":"ContainerStarted","Data":"e223deaf9fa81d83d90df515ee834c6cd37917598c6af3d77825ee5d6def6385"}
Sep 17 18:39:10.966891 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.966860    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f" event={"ID":"0d990377-23e1-48f8-8574-bca87f71c957","Type":"ContainerStarted","Data":"c0c11e0b9599c17ecd78dccbe68738a82a18d9f01a42acd95b886890522f6c0f"}
Sep 17 18:39:10.971607 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.970482    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:39:10.971607 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.971017    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerStarted","Data":"64d30d86e221c1a264977c418d8ae698e6f135abf7b33b6f0573c235db9223b4"}
Sep 17 18:39:10.982142 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.982098    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jnmf8" podStartSLOduration=4.555764265 podStartE2EDuration="21.982084167s" totalImagesPullingTime="17.426319902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:10.08970743 +0000 UTC m=+23.444966204" watchObservedRunningTime="2026-09-17 18:39:10.982084167 +0000 UTC m=+24.337342940"
Sep 17 18:39:10.982352 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:10.982331    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7wll2" podStartSLOduration=4.43991145 podStartE2EDuration="21.982323237s" totalImagesPullingTime="17.542411787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:10.981974089 +0000 UTC m=+24.337232882" watchObservedRunningTime="2026-09-17 18:39:10.982323237 +0000 UTC m=+24.337582010"
Sep 17 18:39:11.554290 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:11.554258    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:39:11.554981 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:11.554959    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:39:11.791337 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:11.791196    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:39:10.81641646Z","UUID":"6a93ce90-77bd-4e5e-a1ba-9075857ba2f9","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:39:11.792896 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:11.792772    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:39:11.792896 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:11.792813    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:39:11.830326 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:11.830304    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:11.830481 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:11.830414    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:11.830554 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:11.830410    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:11.830629 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:11.830546    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:11.977607 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:11.977456    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:39:11.978187 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:11.978172    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ns2kp"
Sep 17 18:39:12.829911 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:12.829881    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:12.830085 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:12.830011    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:12.978791 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:12.978758    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:39:12.979260 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:12.979235    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerStarted","Data":"67cd3fe12256ce061b7514c89776c839064b3631e4669d3fdbffd0a38c67dd67"}
Sep 17 18:39:12.981613 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:12.981586    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f" event={"ID":"0d990377-23e1-48f8-8574-bca87f71c957","Type":"ContainerStarted","Data":"4ec0178730bb56bd6d51d70e29e79e19b8174ec7bd6547b6105b542df55fbdbf"}
Sep 17 18:39:13.003892 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:13.003834    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc4f" podStartSLOduration=4.239448475 podStartE2EDuration="24.003818917s" totalImagesPullingTime="19.764370442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:13.003786969 +0000 UTC m=+26.359045743" watchObservedRunningTime="2026-09-17 18:39:13.003818917 +0000 UTC m=+26.359077689"
Sep 17 18:39:13.830482 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:13.830448    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:13.830640 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:13.830459    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:13.830640 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:13.830566    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:13.830727 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:13.830668    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:14.830859 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:14.830599    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:14.831561 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:14.830973    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:14.989435 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:14.989410    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:39:14.989783 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:14.989759    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerStarted","Data":"f2be667aa5d3b4e4096e27563111fe86e24237b78db44402ac0ed4d3674de8c8"}
Sep 17 18:39:14.990026 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:14.990002    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:39:14.990197 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:14.990179    2557 scope.go:117] "RemoveContainer" containerID="8438db538faabf428ecf304428580fc4cb4794bf2aa73ce523b999744f453aaf"
Sep 17 18:39:14.991627 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:14.991607    2557 generic.go:356] "Generic (PLEG): container finished" podID="9bbd018c-8b15-4a81-a4b6-b96685c1afff" containerID="95dbc3aaef692ef6e924ffe791abbb858951d7aa5bb1d66d1b27616b491d5a00" exitCode=0
Sep 17 18:39:14.991698 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:14.991652    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wl4cz" event={"ID":"9bbd018c-8b15-4a81-a4b6-b96685c1afff","Type":"ContainerDied","Data":"95dbc3aaef692ef6e924ffe791abbb858951d7aa5bb1d66d1b27616b491d5a00"}
Sep 17 18:39:15.006738 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:15.006710    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:39:15.830258 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:15.830227    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:15.830393 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:15.830233    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:15.830393 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:15.830334    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:15.830464 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:15.830426    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:15.998258 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:15.998164    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:39:15.999223 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:15.999173    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" event={"ID":"98765f76-dc15-47c1-ab00-28533582d24b","Type":"ContainerStarted","Data":"01ab9687e92f8d29b187d6312e0333e148e766eb9844203caad55a39fa732eb5"}
Sep 17 18:39:15.999482 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:15.999463    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:39:15.999580 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:15.999537    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:39:16.001635 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.001611    2557 generic.go:356] "Generic (PLEG): container finished" podID="9bbd018c-8b15-4a81-a4b6-b96685c1afff" containerID="c503503d34cecaa652691d59013aafd9ea6ceca4597953f5d2ac5b71e1194b51" exitCode=0
Sep 17 18:39:16.001750 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.001655    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wl4cz" event={"ID":"9bbd018c-8b15-4a81-a4b6-b96685c1afff","Type":"ContainerDied","Data":"c503503d34cecaa652691d59013aafd9ea6ceca4597953f5d2ac5b71e1194b51"}
Sep 17 18:39:16.015443 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.015414    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:39:16.032116 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.032068    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7" podStartSLOduration=9.603431543 podStartE2EDuration="27.032055188s" totalImagesPullingTime="17.428623645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:16.031582198 +0000 UTC m=+29.386840999" watchObservedRunningTime="2026-09-17 18:39:16.032055188 +0000 UTC m=+29.387313988"
Sep 17 18:39:16.554387 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.554170    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x7wf6"]
Sep 17 18:39:16.554552 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.554466    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:16.554626 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:16.554571    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:16.556602 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.556580    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-thgrv"]
Sep 17 18:39:16.556729 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.556676    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:16.556787 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:16.556772    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:16.578631 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.578608    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvr8b"]
Sep 17 18:39:16.578718 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:16.578684    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:16.578795 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:16.578766    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:17.006432 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:17.006401    2557 generic.go:356] "Generic (PLEG): container finished" podID="9bbd018c-8b15-4a81-a4b6-b96685c1afff" containerID="bb0b96e46a54a516de364594f0631a60ba5a0f682f72281a9b05112bada0c051" exitCode=0
Sep 17 18:39:17.006784 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:17.006490    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wl4cz" event={"ID":"9bbd018c-8b15-4a81-a4b6-b96685c1afff","Type":"ContainerDied","Data":"bb0b96e46a54a516de364594f0631a60ba5a0f682f72281a9b05112bada0c051"}
Sep 17 18:39:17.829958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:17.829925    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:17.829958 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:17.829956    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:17.830177 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:17.829956    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:17.830177 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:17.830065    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:17.830177 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:17.830134    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:17.830318 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:17.830232    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:19.830601 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:19.830563    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:19.831108 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:19.830618    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:19.831108 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:19.830575    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:19.831108 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:19.830697    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:19.831108 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:19.830784    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:19.831108 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:19.830868    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:21.830538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:21.830508    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:21.830997 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:21.830545    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:21.830997 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:21.830558    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:21.830997 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:21.830633    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-thgrv" podUID="fc00a436-36cb-41da-b35c-c63466007e96"
Sep 17 18:39:21.830997 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:21.830719    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7wf6" podUID="19f4f176-cbe7-4e1b-bb05-d7483eefe906"
Sep 17 18:39:21.830997 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:21.830791    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvr8b" podUID="a0f20d3b-b7f6-435b-9375-bef4013e5de5"
Sep 17 18:39:22.021896 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.021861    2557 generic.go:356] "Generic (PLEG): container finished" podID="9bbd018c-8b15-4a81-a4b6-b96685c1afff" containerID="160f4c142f0ff43ae4a469e11975da60dbf4b589691bd7f8479287b66d0ca6f3" exitCode=0
Sep 17 18:39:22.022092 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.021903    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wl4cz" event={"ID":"9bbd018c-8b15-4a81-a4b6-b96685c1afff","Type":"ContainerDied","Data":"160f4c142f0ff43ae4a469e11975da60dbf4b589691bd7f8479287b66d0ca6f3"}
Sep 17 18:39:22.171127 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.170968    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-5.ec2.internal" event="NodeReady"
Sep 17 18:39:22.171276 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.171250    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:39:22.213251 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.213202    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f4d48d475-4rxt2"]
Sep 17 18:39:22.215891 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.215876    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.218773 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.218755    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:39:22.219311 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.219299    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:39:22.219612 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.219600    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:39:22.219880 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.219864    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6grnc\""
Sep 17 18:39:22.229534 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.229512    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:39:22.232465 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.232445    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4d48d475-4rxt2"]
Sep 17 18:39:22.252911 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.252889    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p4xzv"]
Sep 17 18:39:22.255598 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.255584    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.259037 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.259020    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9x8j\""
Sep 17 18:39:22.259037 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.259030    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:39:22.259247 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.259229    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:39:22.264044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.264026    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zzl8m"]
Sep 17 18:39:22.266564 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.266551    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zzl8m"
Sep 17 18:39:22.269614 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.269589    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:39:22.269711 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.269621    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:39:22.269711 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.269641    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5pvxj\""
Sep 17 18:39:22.269879 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.269856    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p4xzv"]
Sep 17 18:39:22.269979 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.269963    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:39:22.284708 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.284678    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zzl8m"]
Sep 17 18:39:22.335237 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6124ed7a-8ab2-4762-a1fd-a676936651b2-ca-trust-extracted\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.335237 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335204    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-trusted-ca\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.335353 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335246    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4494cc42-c46b-47e9-a716-42b1777a1e75-config-volume\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.335353 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335296    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.335353 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-image-registry-private-configuration\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.335441 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335348    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4494cc42-c46b-47e9-a716-42b1777a1e75-tmp-dir\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.335441 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335421    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c93cb7fc-25e5-42fe-8050-7079316270d5-cert\") pod \"ingress-canary-zzl8m\" (UID: \"c93cb7fc-25e5-42fe-8050-7079316270d5\") " pod="openshift-ingress-canary/ingress-canary-zzl8m"
Sep 17 18:39:22.335502 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335445    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7zrd\" (UniqueName: \"kubernetes.io/projected/c93cb7fc-25e5-42fe-8050-7079316270d5-kube-api-access-r7zrd\") pod \"ingress-canary-zzl8m\" (UID: \"c93cb7fc-25e5-42fe-8050-7079316270d5\") " pod="openshift-ingress-canary/ingress-canary-zzl8m"
Sep 17 18:39:22.335502 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335464    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbc5\" (UniqueName: \"kubernetes.io/projected/4494cc42-c46b-47e9-a716-42b1777a1e75-kube-api-access-5zbc5\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.335502 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335483    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgkk\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-kube-api-access-sqgkk\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.335589 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335508    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-installation-pull-secrets\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.335589 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335553    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-bound-sa-token\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.335650 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.335650 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.335630    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-certificates\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.436114 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c93cb7fc-25e5-42fe-8050-7079316270d5-cert\") pod \"ingress-canary-zzl8m\" (UID: \"c93cb7fc-25e5-42fe-8050-7079316270d5\") " pod="openshift-ingress-canary/ingress-canary-zzl8m"
Sep 17 18:39:22.436114 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436119    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7zrd\" (UniqueName: \"kubernetes.io/projected/c93cb7fc-25e5-42fe-8050-7079316270d5-kube-api-access-r7zrd\") pod \"ingress-canary-zzl8m\" (UID: \"c93cb7fc-25e5-42fe-8050-7079316270d5\") " pod="openshift-ingress-canary/ingress-canary-zzl8m"
Sep 17 18:39:22.436388 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbc5\" (UniqueName: \"kubernetes.io/projected/4494cc42-c46b-47e9-a716-42b1777a1e75-kube-api-access-5zbc5\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.436388 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436170    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgkk\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-kube-api-access-sqgkk\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.436388 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436198    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-installation-pull-secrets\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.436388 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436241    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-bound-sa-token\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.436388 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436267    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.436388 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436292    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-certificates\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.436388 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6124ed7a-8ab2-4762-a1fd-a676936651b2-ca-trust-extracted\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.436388 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-trusted-ca\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.436754 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436414    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4494cc42-c46b-47e9-a716-42b1777a1e75-config-volume\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.436754 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.436754 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.436450    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:22.436754 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.436469    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d48d475-4rxt2: secret "image-registry-tls" not found
Sep 17 18:39:22.436754 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436466    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-image-registry-private-configuration\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.436754 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4494cc42-c46b-47e9-a716-42b1777a1e75-tmp-dir\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.436754 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.436534    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls podName:6124ed7a-8ab2-4762-a1fd-a676936651b2 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:22.936500967 +0000 UTC m=+36.291759734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls") pod "image-registry-7f4d48d475-4rxt2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2") : secret "image-registry-tls" not found
Sep 17 18:39:22.437092 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436825    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4494cc42-c46b-47e9-a716-42b1777a1e75-tmp-dir\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.437092 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.436914    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6124ed7a-8ab2-4762-a1fd-a676936651b2-ca-trust-extracted\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.437092 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.437013    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:22.437092 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.437023    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-certificates\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.437092 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.437054    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls podName:4494cc42-c46b-47e9-a716-42b1777a1e75 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:22.937040414 +0000 UTC m=+36.292299179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls") pod "dns-default-p4xzv" (UID: "4494cc42-c46b-47e9-a716-42b1777a1e75") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:22.437347 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.437324    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-trusted-ca\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.437394 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.437375    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4494cc42-c46b-47e9-a716-42b1777a1e75-config-volume\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.440164 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.440141    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-image-registry-private-configuration\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.440164 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.440163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-installation-pull-secrets\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.440330 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.440165    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c93cb7fc-25e5-42fe-8050-7079316270d5-cert\") pod \"ingress-canary-zzl8m\" (UID: \"c93cb7fc-25e5-42fe-8050-7079316270d5\") " pod="openshift-ingress-canary/ingress-canary-zzl8m"
Sep 17 18:39:22.446933 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.446910    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7zrd\" (UniqueName: \"kubernetes.io/projected/c93cb7fc-25e5-42fe-8050-7079316270d5-kube-api-access-r7zrd\") pod \"ingress-canary-zzl8m\" (UID: \"c93cb7fc-25e5-42fe-8050-7079316270d5\") " pod="openshift-ingress-canary/ingress-canary-zzl8m"
Sep 17 18:39:22.454579 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.454552    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-bound-sa-token\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.454681 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.454593    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbc5\" (UniqueName: \"kubernetes.io/projected/4494cc42-c46b-47e9-a716-42b1777a1e75-kube-api-access-5zbc5\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.454681 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.454637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgkk\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-kube-api-access-sqgkk\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.455443 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.455420    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ns2kp_3b22214b-faa3-490b-8e11-2ea7b4e6e15f/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:22.537615 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.537577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:22.537757 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.537642    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:22.537757 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.537724    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:22.537824 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.537783    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret podName:fc00a436-36cb-41da-b35c-c63466007e96 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:54.537767886 +0000 UTC m=+67.893026641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret") pod "global-pull-secret-syncer-thgrv" (UID: "fc00a436-36cb-41da-b35c-c63466007e96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:39:22.537824 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.537730    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:22.537897 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.537862    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs podName:a0f20d3b-b7f6-435b-9375-bef4013e5de5 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:54.537846952 +0000 UTC m=+67.893105703 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs") pod "network-metrics-daemon-qvr8b" (UID: "a0f20d3b-b7f6-435b-9375-bef4013e5de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:39:22.569582 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.569524    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ns2kp_3b22214b-faa3-490b-8e11-2ea7b4e6e15f/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:22.575459 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.575438    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zzl8m"
Sep 17 18:39:22.638970 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.638941    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:22.639373 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.639350    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:39:22.639478 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.639379    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:39:22.639478 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.639392    2557 projected.go:196] Error preparing data for projected volume kube-api-access-jg4r9 for pod openshift-network-diagnostics/network-check-target-x7wf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:22.639478 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.639457    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9 podName:19f4f176-cbe7-4e1b-bb05-d7483eefe906 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:54.639439095 +0000 UTC m=+67.994697862 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jg4r9" (UniqueName: "kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9") pod "network-check-target-x7wf6" (UID: "19f4f176-cbe7-4e1b-bb05-d7483eefe906") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:39:22.856941 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.856862    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zzl8m"]
Sep 17 18:39:22.860283 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:39:22.860250    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93cb7fc_25e5_42fe_8050_7079316270d5.slice/crio-e182ead62cf169633fa0624ba1d3aa9958b1f4fd65e5987c3a646ca318afb985 WatchSource:0}: Error finding container e182ead62cf169633fa0624ba1d3aa9958b1f4fd65e5987c3a646ca318afb985: Status 404 returned error can't find the container with id e182ead62cf169633fa0624ba1d3aa9958b1f4fd65e5987c3a646ca318afb985
Sep 17 18:39:22.941553 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.941515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:22.941738 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:22.941576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:22.941738 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.941672    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:22.941738 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.941716    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls podName:4494cc42-c46b-47e9-a716-42b1777a1e75 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:23.94170404 +0000 UTC m=+37.296962791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls") pod "dns-default-p4xzv" (UID: "4494cc42-c46b-47e9-a716-42b1777a1e75") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:22.941738 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.941671    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:22.941738 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.941734    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d48d475-4rxt2: secret "image-registry-tls" not found
Sep 17 18:39:22.941914 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:22.941755    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls podName:6124ed7a-8ab2-4762-a1fd-a676936651b2 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:23.941747809 +0000 UTC m=+37.297006560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls") pod "image-registry-7f4d48d475-4rxt2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2") : secret "image-registry-tls" not found
Sep 17 18:39:23.027098 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.027069    2557 generic.go:356] "Generic (PLEG): container finished" podID="9bbd018c-8b15-4a81-a4b6-b96685c1afff" containerID="7ccc30017ef5e2851a6d2229d2bfee38abacae8369bdcad589dd21917079946e" exitCode=0
Sep 17 18:39:23.027291 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.027154    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wl4cz" event={"ID":"9bbd018c-8b15-4a81-a4b6-b96685c1afff","Type":"ContainerDied","Data":"7ccc30017ef5e2851a6d2229d2bfee38abacae8369bdcad589dd21917079946e"}
Sep 17 18:39:23.028322 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.028288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zzl8m" event={"ID":"c93cb7fc-25e5-42fe-8050-7079316270d5","Type":"ContainerStarted","Data":"e182ead62cf169633fa0624ba1d3aa9958b1f4fd65e5987c3a646ca318afb985"}
Sep 17 18:39:23.830714 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.830484    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:23.830891 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.830485    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:23.830891 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.830523    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:23.835277 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.835254    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:39:23.835277 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.835267    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nghzc\""
Sep 17 18:39:23.835459 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.835257    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-shgtw\""
Sep 17 18:39:23.835612 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.835552    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:39:23.835612 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.835571    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:39:23.835815 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.835801    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:39:23.948574 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.948542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:23.949035 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:23.948627    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:23.949035 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:23.948724    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:23.949035 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:23.948796    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls podName:4494cc42-c46b-47e9-a716-42b1777a1e75 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:25.948774903 +0000 UTC m=+39.304033656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls") pod "dns-default-p4xzv" (UID: "4494cc42-c46b-47e9-a716-42b1777a1e75") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:23.949035 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:23.948728    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:23.949035 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:23.948850    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d48d475-4rxt2: secret "image-registry-tls" not found
Sep 17 18:39:23.949035 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:23.948911    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls podName:6124ed7a-8ab2-4762-a1fd-a676936651b2 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:25.948893342 +0000 UTC m=+39.304152109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls") pod "image-registry-7f4d48d475-4rxt2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2") : secret "image-registry-tls" not found
Sep 17 18:39:24.034726 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:24.034694    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wl4cz" event={"ID":"9bbd018c-8b15-4a81-a4b6-b96685c1afff","Type":"ContainerStarted","Data":"04b0bc3fdb40b2d91d6aeeffa6063aded707e3d76989e56ff47d3e406abd642c"}
Sep 17 18:39:24.060576 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:24.060528    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wl4cz" podStartSLOduration=7.915842046 podStartE2EDuration="36.060510747s" totalImagesPullingTime="28.144668701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:24.058993406 +0000 UTC m=+37.414252199" watchObservedRunningTime="2026-09-17 18:39:24.060510747 +0000 UTC m=+37.415769519"
Sep 17 18:39:24.183115 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:24.183024    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ns2kp_3b22214b-faa3-490b-8e11-2ea7b4e6e15f/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:24.283788 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:24.283741    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ns2kp_3b22214b-faa3-490b-8e11-2ea7b4e6e15f/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:24.676528 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:24.676487    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ns2kp_3b22214b-faa3-490b-8e11-2ea7b4e6e15f/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:24.752441 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:24.752405    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ns2kp_3b22214b-faa3-490b-8e11-2ea7b4e6e15f/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:25.964863 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:25.964781    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:25.964863 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:25.964844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:25.965335 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:25.964936    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:25.965335 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:25.964945    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:25.965335 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:25.964964    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d48d475-4rxt2: secret "image-registry-tls" not found
Sep 17 18:39:25.965335 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:25.964987    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls podName:4494cc42-c46b-47e9-a716-42b1777a1e75 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:29.964974637 +0000 UTC m=+43.320233402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls") pod "dns-default-p4xzv" (UID: "4494cc42-c46b-47e9-a716-42b1777a1e75") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:25.965335 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:25.965010    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls podName:6124ed7a-8ab2-4762-a1fd-a676936651b2 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:29.964996833 +0000 UTC m=+43.320255584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls") pod "image-registry-7f4d48d475-4rxt2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2") : secret "image-registry-tls" not found
Sep 17 18:39:26.041200 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:26.041162    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zzl8m" event={"ID":"c93cb7fc-25e5-42fe-8050-7079316270d5","Type":"ContainerStarted","Data":"951faf241d2112e491f4d7b1c8c38b48234cdde4a3a49407ff3e5d1cc98c00ce"}
Sep 17 18:39:29.996320 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:29.996274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:29.996750 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:29.996368    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:29.996750 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:29.996407    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:29.996750 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:29.996429    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d48d475-4rxt2: secret "image-registry-tls" not found
Sep 17 18:39:29.996750 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:29.996491    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls podName:6124ed7a-8ab2-4762-a1fd-a676936651b2 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:37.996474794 +0000 UTC m=+51.351733549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls") pod "image-registry-7f4d48d475-4rxt2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2") : secret "image-registry-tls" not found
Sep 17 18:39:29.996750 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:29.996503    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:29.996750 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:29.996549    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls podName:4494cc42-c46b-47e9-a716-42b1777a1e75 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:37.996536615 +0000 UTC m=+51.351795385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls") pod "dns-default-p4xzv" (UID: "4494cc42-c46b-47e9-a716-42b1777a1e75") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:38.051040 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:38.050941    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:38.051040 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:38.051010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:38.051482 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:38.051102    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:38.051482 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:38.051102    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:38.051482 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:38.051180    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls podName:4494cc42-c46b-47e9-a716-42b1777a1e75 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:54.051164906 +0000 UTC m=+67.406423656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls") pod "dns-default-p4xzv" (UID: "4494cc42-c46b-47e9-a716-42b1777a1e75") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:38.051482 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:38.051113    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d48d475-4rxt2: secret "image-registry-tls" not found
Sep 17 18:39:38.051482 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:38.051272    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls podName:6124ed7a-8ab2-4762-a1fd-a676936651b2 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:54.051259667 +0000 UTC m=+67.406518418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls") pod "image-registry-7f4d48d475-4rxt2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2") : secret "image-registry-tls" not found
Sep 17 18:39:48.020196 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:48.020166    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z6rd7"
Sep 17 18:39:48.051968 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:48.051911    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zzl8m" podStartSLOduration=23.298955549 podStartE2EDuration="26.051896273s" totalImagesPullingTime="2.752940724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:26.082010443 +0000 UTC m=+39.437269234" watchObservedRunningTime="2026-09-17 18:39:48.051896273 +0000 UTC m=+61.407155083"
Sep 17 18:39:54.067477 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.067434    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:39:54.067980 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.067499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:39:54.067980 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:54.067583    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:39:54.067980 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:54.067593    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:39:54.067980 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:54.067614    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f4d48d475-4rxt2: secret "image-registry-tls" not found
Sep 17 18:39:54.067980 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:54.067646    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls podName:4494cc42-c46b-47e9-a716-42b1777a1e75 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:26.067632752 +0000 UTC m=+99.422891503 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls") pod "dns-default-p4xzv" (UID: "4494cc42-c46b-47e9-a716-42b1777a1e75") : secret "dns-default-metrics-tls" not found
Sep 17 18:39:54.067980 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:54.067667    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls podName:6124ed7a-8ab2-4762-a1fd-a676936651b2 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:26.067653536 +0000 UTC m=+99.422912287 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls") pod "image-registry-7f4d48d475-4rxt2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2") : secret "image-registry-tls" not found
Sep 17 18:39:54.571968 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.571934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:54.572112 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.571998    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:39:54.575076 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.575058    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:39:54.575126 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.575063    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:39:54.582621 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:54.582604    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 18:39:54.582685 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:39:54.582675    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs podName:a0f20d3b-b7f6-435b-9375-bef4013e5de5 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:58.58265452 +0000 UTC m=+131.937913289 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs") pod "network-metrics-daemon-qvr8b" (UID: "a0f20d3b-b7f6-435b-9375-bef4013e5de5") : secret "metrics-daemon-secret" not found
Sep 17 18:39:54.585747 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.585725    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc00a436-36cb-41da-b35c-c63466007e96-original-pull-secret\") pod \"global-pull-secret-syncer-thgrv\" (UID: \"fc00a436-36cb-41da-b35c-c63466007e96\") " pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:54.672527 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.672480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:54.675843 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.675823    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:39:54.686191 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.686170    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:39:54.695867 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.695851    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4r9\" (UniqueName: \"kubernetes.io/projected/19f4f176-cbe7-4e1b-bb05-d7483eefe906-kube-api-access-jg4r9\") pod \"network-check-target-x7wf6\" (UID: \"19f4f176-cbe7-4e1b-bb05-d7483eefe906\") " pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:54.752160 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.752132    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-shgtw\""
Sep 17 18:39:54.754168 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.754152    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-thgrv"
Sep 17 18:39:54.759289 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:54.759259    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:39:55.035227 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:55.035178    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-thgrv"]
Sep 17 18:39:55.038070 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:55.038046    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x7wf6"]
Sep 17 18:39:55.039243 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:39:55.039200    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc00a436_36cb_41da_b35c_c63466007e96.slice/crio-771f187b156ae143af2fe25bbed964b01c469175ddc62f110edc4c940be9951b WatchSource:0}: Error finding container 771f187b156ae143af2fe25bbed964b01c469175ddc62f110edc4c940be9951b: Status 404 returned error can't find the container with id 771f187b156ae143af2fe25bbed964b01c469175ddc62f110edc4c940be9951b
Sep 17 18:39:55.040635 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:39:55.040617    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f4f176_cbe7_4e1b_bb05_d7483eefe906.slice/crio-1ab8690da595f41b0262448575e337af55460da4556afa11c6cc81eb492f6d1e WatchSource:0}: Error finding container 1ab8690da595f41b0262448575e337af55460da4556afa11c6cc81eb492f6d1e: Status 404 returned error can't find the container with id 1ab8690da595f41b0262448575e337af55460da4556afa11c6cc81eb492f6d1e
Sep 17 18:39:55.108678 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:55.108598    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x7wf6" event={"ID":"19f4f176-cbe7-4e1b-bb05-d7483eefe906","Type":"ContainerStarted","Data":"1ab8690da595f41b0262448575e337af55460da4556afa11c6cc81eb492f6d1e"}
Sep 17 18:39:55.109633 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:39:55.109605    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-thgrv" event={"ID":"fc00a436-36cb-41da-b35c-c63466007e96","Type":"ContainerStarted","Data":"771f187b156ae143af2fe25bbed964b01c469175ddc62f110edc4c940be9951b"}
Sep 17 18:40:00.126186 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:00.126147    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x7wf6" event={"ID":"19f4f176-cbe7-4e1b-bb05-d7483eefe906","Type":"ContainerStarted","Data":"b83ef8e86ed851e8fd2547e9568cb67756db4e69b261fd5776c890e5a1b9bbbe"}
Sep 17 18:40:00.126659 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:00.126278    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:40:00.127740 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:00.127718    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-thgrv" event={"ID":"fc00a436-36cb-41da-b35c-c63466007e96","Type":"ContainerStarted","Data":"859f32f10571d7b5ddea7dd423044d645a5975e309f308a33b433162f6ec3844"}
Sep 17 18:40:00.143859 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:00.143820    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x7wf6" podStartSLOduration=66.463012197 podStartE2EDuration="1m11.14380687s" totalImagesPullingTime="4.680794673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:00.142844338 +0000 UTC m=+73.498103110" watchObservedRunningTime="2026-09-17 18:40:00.14380687 +0000 UTC m=+73.499065640"
Sep 17 18:40:00.161482 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:00.161434    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-thgrv" podStartSLOduration=66.480213774 podStartE2EDuration="1m11.161419341s" totalImagesPullingTime="4.681205567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:00.160807796 +0000 UTC m=+73.516066569" watchObservedRunningTime="2026-09-17 18:40:00.161419341 +0000 UTC m=+73.516678113"
Sep 17 18:40:03.532635 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.532500    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f"]
Sep 17 18:40:03.536397 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.536381    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-pnx6j"]
Sep 17 18:40:03.536517 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.536501    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f"
Sep 17 18:40:03.539455 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.539436    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-j52sk\""
Sep 17 18:40:03.539858 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.539841    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.543223 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.543189    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:40:03.543543 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.543525    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 18:40:03.543645 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.543623    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 18:40:03.543710 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.543648    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vrnwg\""
Sep 17 18:40:03.544488 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.544471    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 18:40:03.551137 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.551116    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f"]
Sep 17 18:40:03.553167 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.553148    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 18:40:03.557083 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.557061    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-pnx6j"]
Sep 17 18:40:03.632183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.632153    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c74d2858-9009-45c9-b7c5-e2ade2bb6311-trusted-ca\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.632365 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.632203    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82cc\" (UniqueName: \"kubernetes.io/projected/38340984-1048-4349-93ae-211944b75b16-kube-api-access-h82cc\") pod \"network-check-source-7cbc45fddc-m7l6f\" (UID: \"38340984-1048-4349-93ae-211944b75b16\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f"
Sep 17 18:40:03.632365 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.632273    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c74d2858-9009-45c9-b7c5-e2ade2bb6311-config\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.632365 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.632309    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c74d2858-9009-45c9-b7c5-e2ade2bb6311-serving-cert\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.632365 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.632362    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h684x\" (UniqueName: \"kubernetes.io/projected/c74d2858-9009-45c9-b7c5-e2ade2bb6311-kube-api-access-h684x\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.720022 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.719990    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"]
Sep 17 18:40:03.723752 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.723729    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:03.724121 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.724094    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"]
Sep 17 18:40:03.726729 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.726705    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 18:40:03.727194 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.727176    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 18:40:03.727194 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.727186    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:40:03.727415 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.727402    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-r5v7f\""
Sep 17 18:40:03.727459 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.727437    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 18:40:03.727573 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.727557    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:03.731872 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.731853    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 18:40:03.731975 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.731876    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xh9jx\""
Sep 17 18:40:03.731975 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.731949    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 18:40:03.732197 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.732182    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:40:03.733028 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.733007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h684x\" (UniqueName: \"kubernetes.io/projected/c74d2858-9009-45c9-b7c5-e2ade2bb6311-kube-api-access-h684x\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.733097 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.733047    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c74d2858-9009-45c9-b7c5-e2ade2bb6311-trusted-ca\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.733097 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.733090    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82cc\" (UniqueName: \"kubernetes.io/projected/38340984-1048-4349-93ae-211944b75b16-kube-api-access-h82cc\") pod \"network-check-source-7cbc45fddc-m7l6f\" (UID: \"38340984-1048-4349-93ae-211944b75b16\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f"
Sep 17 18:40:03.733196 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.733123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c74d2858-9009-45c9-b7c5-e2ade2bb6311-config\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.733196 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.733161    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c74d2858-9009-45c9-b7c5-e2ade2bb6311-serving-cert\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.733773 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.733755    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c74d2858-9009-45c9-b7c5-e2ade2bb6311-config\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.733840 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.733812    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c74d2858-9009-45c9-b7c5-e2ade2bb6311-trusted-ca\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.735717 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.735697    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c74d2858-9009-45c9-b7c5-e2ade2bb6311-serving-cert\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.740476 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.740454    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"]
Sep 17 18:40:03.746576 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.746548    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82cc\" (UniqueName: \"kubernetes.io/projected/38340984-1048-4349-93ae-211944b75b16-kube-api-access-h82cc\") pod \"network-check-source-7cbc45fddc-m7l6f\" (UID: \"38340984-1048-4349-93ae-211944b75b16\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f"
Sep 17 18:40:03.749324 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.749306    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h684x\" (UniqueName: \"kubernetes.io/projected/c74d2858-9009-45c9-b7c5-e2ade2bb6311-kube-api-access-h684x\") pod \"console-operator-fd4bb46c-pnx6j\" (UID: \"c74d2858-9009-45c9-b7c5-e2ade2bb6311\") " pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.749613 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.749594    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"]
Sep 17 18:40:03.834233 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.834174    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb4df\" (UniqueName: \"kubernetes.io/projected/3eaf0419-6fce-4618-8075-efbf565b392b-kube-api-access-sb4df\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:03.834414 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.834296    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:03.834414 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.834326    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4hx\" (UniqueName: \"kubernetes.io/projected/5f273124-4276-4588-962a-4424f23b76b8-kube-api-access-fs4hx\") pod \"kube-storage-version-migrator-operator-55889f7b47-vv6wk\" (UID: \"5f273124-4276-4588-962a-4424f23b76b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:03.834414 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.834374    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f273124-4276-4588-962a-4424f23b76b8-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-vv6wk\" (UID: \"5f273124-4276-4588-962a-4424f23b76b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:03.834532 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.834420    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f273124-4276-4588-962a-4424f23b76b8-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-vv6wk\" (UID: \"5f273124-4276-4588-962a-4424f23b76b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:03.846178 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.846154    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f"
Sep 17 18:40:03.850873 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.850853    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:03.935400 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.935372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4df\" (UniqueName: \"kubernetes.io/projected/3eaf0419-6fce-4618-8075-efbf565b392b-kube-api-access-sb4df\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:03.935545 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.935436    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:03.935545 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.935464    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4hx\" (UniqueName: \"kubernetes.io/projected/5f273124-4276-4588-962a-4424f23b76b8-kube-api-access-fs4hx\") pod \"kube-storage-version-migrator-operator-55889f7b47-vv6wk\" (UID: \"5f273124-4276-4588-962a-4424f23b76b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:03.935545 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.935500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f273124-4276-4588-962a-4424f23b76b8-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-vv6wk\" (UID: \"5f273124-4276-4588-962a-4424f23b76b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:03.935545 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.935541    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f273124-4276-4588-962a-4424f23b76b8-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-vv6wk\" (UID: \"5f273124-4276-4588-962a-4424f23b76b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:03.935982 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:03.935959    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:03.936058 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:03.936024    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls podName:3eaf0419-6fce-4618-8075-efbf565b392b nodeName:}" failed. No retries permitted until 2026-09-17 18:40:04.436009787 +0000 UTC m=+77.791268538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-vprf5" (UID: "3eaf0419-6fce-4618-8075-efbf565b392b") : secret "samples-operator-tls" not found
Sep 17 18:40:03.936124 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.936088    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5f273124-4276-4588-962a-4424f23b76b8-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-vv6wk\" (UID: \"5f273124-4276-4588-962a-4424f23b76b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:03.938259 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.938236    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5f273124-4276-4588-962a-4424f23b76b8-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-vv6wk\" (UID: \"5f273124-4276-4588-962a-4424f23b76b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:03.945363 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.945341    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb4df\" (UniqueName: \"kubernetes.io/projected/3eaf0419-6fce-4618-8075-efbf565b392b-kube-api-access-sb4df\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:03.945577 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:03.945558    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4hx\" (UniqueName: \"kubernetes.io/projected/5f273124-4276-4588-962a-4424f23b76b8-kube-api-access-fs4hx\") pod \"kube-storage-version-migrator-operator-55889f7b47-vv6wk\" (UID: \"5f273124-4276-4588-962a-4424f23b76b8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:04.033840 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:04.033803    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"
Sep 17 18:40:04.121461 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:04.121436    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f"]
Sep 17 18:40:04.123741 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:40:04.123713    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38340984_1048_4349_93ae_211944b75b16.slice/crio-b41bdbf7c694aaae6c34a52e861c601c887c031b851c21ebf3de7bb67436a3f1 WatchSource:0}: Error finding container b41bdbf7c694aaae6c34a52e861c601c887c031b851c21ebf3de7bb67436a3f1: Status 404 returned error can't find the container with id b41bdbf7c694aaae6c34a52e861c601c887c031b851c21ebf3de7bb67436a3f1
Sep 17 18:40:04.127507 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:04.127485    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-pnx6j"]
Sep 17 18:40:04.129794 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:40:04.129775    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74d2858_9009_45c9_b7c5_e2ade2bb6311.slice/crio-af8ab47d092e9107c2cdf215630f3982d4519276d7ec2b7ca933bd099f0abb6c WatchSource:0}: Error finding container af8ab47d092e9107c2cdf215630f3982d4519276d7ec2b7ca933bd099f0abb6c: Status 404 returned error can't find the container with id af8ab47d092e9107c2cdf215630f3982d4519276d7ec2b7ca933bd099f0abb6c
Sep 17 18:40:04.139148 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:04.139125    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f" event={"ID":"38340984-1048-4349-93ae-211944b75b16","Type":"ContainerStarted","Data":"b41bdbf7c694aaae6c34a52e861c601c887c031b851c21ebf3de7bb67436a3f1"}
Sep 17 18:40:04.140144 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:04.140123    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j" event={"ID":"c74d2858-9009-45c9-b7c5-e2ade2bb6311","Type":"ContainerStarted","Data":"af8ab47d092e9107c2cdf215630f3982d4519276d7ec2b7ca933bd099f0abb6c"}
Sep 17 18:40:04.307611 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:04.307586    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk"]
Sep 17 18:40:04.310037 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:40:04.310014    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f273124_4276_4588_962a_4424f23b76b8.slice/crio-a1e79955004ff2d663982c7df2823c5a971bc140360c732bb190295138e23bd1 WatchSource:0}: Error finding container a1e79955004ff2d663982c7df2823c5a971bc140360c732bb190295138e23bd1: Status 404 returned error can't find the container with id a1e79955004ff2d663982c7df2823c5a971bc140360c732bb190295138e23bd1
Sep 17 18:40:04.439567 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:04.439481    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:04.439713 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:04.439631    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:04.439713 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:04.439703    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls podName:3eaf0419-6fce-4618-8075-efbf565b392b nodeName:}" failed. No retries permitted until 2026-09-17 18:40:05.439687861 +0000 UTC m=+78.794946627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-vprf5" (UID: "3eaf0419-6fce-4618-8075-efbf565b392b") : secret "samples-operator-tls" not found
Sep 17 18:40:05.145730 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:05.145659    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk" event={"ID":"5f273124-4276-4588-962a-4424f23b76b8","Type":"ContainerStarted","Data":"a1e79955004ff2d663982c7df2823c5a971bc140360c732bb190295138e23bd1"}
Sep 17 18:40:05.148801 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:05.148253    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f" event={"ID":"38340984-1048-4349-93ae-211944b75b16","Type":"ContainerStarted","Data":"4cbffeccaa9471dceabcf397a05bcdeb6251a670a87b3e9da7d235b8ff4852b5"}
Sep 17 18:40:05.169004 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:05.168952    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-m7l6f" podStartSLOduration=2.168931483 podStartE2EDuration="2.168931483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:05.168858563 +0000 UTC m=+78.524117357" watchObservedRunningTime="2026-09-17 18:40:05.168931483 +0000 UTC m=+78.524190259"
Sep 17 18:40:05.447748 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:05.447661    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:05.447917 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:05.447793    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:05.447917 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:05.447870    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls podName:3eaf0419-6fce-4618-8075-efbf565b392b nodeName:}" failed. No retries permitted until 2026-09-17 18:40:07.447849936 +0000 UTC m=+80.803108691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-vprf5" (UID: "3eaf0419-6fce-4618-8075-efbf565b392b") : secret "samples-operator-tls" not found
Sep 17 18:40:07.156322 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:07.156284    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/0.log"
Sep 17 18:40:07.156762 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:07.156333    2557 generic.go:356] "Generic (PLEG): container finished" podID="c74d2858-9009-45c9-b7c5-e2ade2bb6311" containerID="23f05569f59ba197214413d9f177cfa2f8fcfad47cdf50977652c80c456ab771" exitCode=255
Sep 17 18:40:07.156762 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:07.156431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j" event={"ID":"c74d2858-9009-45c9-b7c5-e2ade2bb6311","Type":"ContainerDied","Data":"23f05569f59ba197214413d9f177cfa2f8fcfad47cdf50977652c80c456ab771"}
Sep 17 18:40:07.156762 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:07.156686    2557 scope.go:117] "RemoveContainer" containerID="23f05569f59ba197214413d9f177cfa2f8fcfad47cdf50977652c80c456ab771"
Sep 17 18:40:07.157968 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:07.157942    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk" event={"ID":"5f273124-4276-4588-962a-4424f23b76b8","Type":"ContainerStarted","Data":"ed67c71c9815da56a5d4bae5e3f4c6c5ec8b20959fe4e125067428eb9dc50b2a"}
Sep 17 18:40:07.197475 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:07.197422    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk" podStartSLOduration=1.926502875 podStartE2EDuration="4.197400599s" totalImagesPullingTime="2.270897724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:07.196663614 +0000 UTC m=+80.551922414" watchObservedRunningTime="2026-09-17 18:40:07.197400599 +0000 UTC m=+80.552659373"
Sep 17 18:40:07.465442 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:07.465345    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:07.465596 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:07.465487    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:07.465596 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:07.465556    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls podName:3eaf0419-6fce-4618-8075-efbf565b392b nodeName:}" failed. No retries permitted until 2026-09-17 18:40:11.465538275 +0000 UTC m=+84.820797025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-vprf5" (UID: "3eaf0419-6fce-4618-8075-efbf565b392b") : secret "samples-operator-tls" not found
Sep 17 18:40:08.162373 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.162345    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:40:08.162766 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.162753    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/0.log"
Sep 17 18:40:08.162813 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.162785    2557 generic.go:356] "Generic (PLEG): container finished" podID="c74d2858-9009-45c9-b7c5-e2ade2bb6311" containerID="a507649592fe78b4a1cd75e2cc8a39c60e43cee0a9fc042c944ccc96a410b7ad" exitCode=255
Sep 17 18:40:08.162870 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.162815    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j" event={"ID":"c74d2858-9009-45c9-b7c5-e2ade2bb6311","Type":"ContainerDied","Data":"a507649592fe78b4a1cd75e2cc8a39c60e43cee0a9fc042c944ccc96a410b7ad"}
Sep 17 18:40:08.162944 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.162931    2557 scope.go:117] "RemoveContainer" containerID="23f05569f59ba197214413d9f177cfa2f8fcfad47cdf50977652c80c456ab771"
Sep 17 18:40:08.163111 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.163098    2557 scope.go:117] "RemoveContainer" containerID="a507649592fe78b4a1cd75e2cc8a39c60e43cee0a9fc042c944ccc96a410b7ad"
Sep 17 18:40:08.163309 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:08.163288    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-pnx6j_openshift-console-operator(c74d2858-9009-45c9-b7c5-e2ade2bb6311)\"" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j" podUID="c74d2858-9009-45c9-b7c5-e2ade2bb6311"
Sep 17 18:40:08.307044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.307006    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cbb2x"]
Sep 17 18:40:08.311351 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.311328    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.315289 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.315268    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:40:08.315599 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.315517    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:40:08.315710 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.315617    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:40:08.315782 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.315755    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vr4zj\""
Sep 17 18:40:08.315963 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.315944    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:40:08.322767 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.322745    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cbb2x"]
Sep 17 18:40:08.371467 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.371438    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-crio-socket\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.371621 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.371478    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.371621 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.371568    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-data-volume\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.371621 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.371595    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkp2h\" (UniqueName: \"kubernetes.io/projected/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-api-access-qkp2h\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.371749 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.371625    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.472172 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.472093    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-data-volume\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.472172 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.472133    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkp2h\" (UniqueName: \"kubernetes.io/projected/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-api-access-qkp2h\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.472172 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.472158    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.472419 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:08.472263    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:08.472419 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.472321    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-crio-socket\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.472419 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:08.472336    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:08.972321909 +0000 UTC m=+82.327580660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:08.472419 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.472385    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.472605 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.472432    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-crio-socket\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.472605 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:08.472452    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:08.472605 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.472462    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-data-volume\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.472605 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:08.472541    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:08.97252247 +0000 UTC m=+82.327781240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:08.499007 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.498973    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkp2h\" (UniqueName: \"kubernetes.io/projected/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-api-access-qkp2h\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.977244 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.977190    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.977469 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:08.977303    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:08.977469 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:08.977333    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:08.977469 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:08.977399    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:09.977385325 +0000 UTC m=+83.332644076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:08.977469 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:08.977401    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:08.977469 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:08.977449    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:09.977435126 +0000 UTC m=+83.332693885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:09.167611 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:09.167586    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:40:09.167962 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:09.167915    2557 scope.go:117] "RemoveContainer" containerID="a507649592fe78b4a1cd75e2cc8a39c60e43cee0a9fc042c944ccc96a410b7ad"
Sep 17 18:40:09.168068 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:09.168049    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-pnx6j_openshift-console-operator(c74d2858-9009-45c9-b7c5-e2ade2bb6311)\"" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j" podUID="c74d2858-9009-45c9-b7c5-e2ade2bb6311"
Sep 17 18:40:09.984928 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:09.984894    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:09.985112 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:09.984959    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:09.985112 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:09.985052    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:09.985112 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:09.985111    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:11.985098809 +0000 UTC m=+85.340357573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:09.985264 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:09.985052    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:09.985264 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:09.985197    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:11.985181938 +0000 UTC m=+85.340440695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:10.053848 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:10.053821    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gdkz_4a59040e-3d81-4961-8cd1-8fc477b7470e/dns-node-resolver/0.log"
Sep 17 18:40:11.050580 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:11.050553    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4947k_f1b9b717-c3af-4187-b781-b6145b367908/node-ca/0.log"
Sep 17 18:40:11.498822 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:11.498784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:11.499001 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:11.498944    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:40:11.499042 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:11.499018    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls podName:3eaf0419-6fce-4618-8075-efbf565b392b nodeName:}" failed. No retries permitted until 2026-09-17 18:40:19.499000208 +0000 UTC m=+92.854258964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-vprf5" (UID: "3eaf0419-6fce-4618-8075-efbf565b392b") : secret "samples-operator-tls" not found
Sep 17 18:40:12.003338 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:12.003301    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:12.003510 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:12.003369    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:12.003510 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:12.003440    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:12.003510 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:12.003446    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:12.003510 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:12.003505    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:16.003486907 +0000 UTC m=+89.358745678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:12.003700 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:12.003519    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:16.003513829 +0000 UTC m=+89.358772580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:12.251302 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:12.251269    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zzl8m_c93cb7fc-25e5-42fe-8050-7079316270d5/serve-healthcheck-canary/0.log"
Sep 17 18:40:13.850961 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:13.850919    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:13.851464 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:13.851034    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:13.851464 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:13.851404    2557 scope.go:117] "RemoveContainer" containerID="a507649592fe78b4a1cd75e2cc8a39c60e43cee0a9fc042c944ccc96a410b7ad"
Sep 17 18:40:13.851613 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:13.851590    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-pnx6j_openshift-console-operator(c74d2858-9009-45c9-b7c5-e2ade2bb6311)\"" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j" podUID="c74d2858-9009-45c9-b7c5-e2ade2bb6311"
Sep 17 18:40:14.184325 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:14.184245    2557 scope.go:117] "RemoveContainer" containerID="a507649592fe78b4a1cd75e2cc8a39c60e43cee0a9fc042c944ccc96a410b7ad"
Sep 17 18:40:14.184454 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:14.184412    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-pnx6j_openshift-console-operator(c74d2858-9009-45c9-b7c5-e2ade2bb6311)\"" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j" podUID="c74d2858-9009-45c9-b7c5-e2ade2bb6311"
Sep 17 18:40:16.034741 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:16.034708    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:16.035120 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:16.034789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:16.035120 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:16.034877    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:16.035120 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:16.034889    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:40:16.035120 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:16.034958    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:24.034932359 +0000 UTC m=+97.390191135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:16.035120 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:16.034984    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:24.03497051 +0000 UTC m=+97.390229261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : configmap "kube-rbac-proxy" not found
Sep 17 18:40:19.559745 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:19.559706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:19.562019 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:19.561998    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3eaf0419-6fce-4618-8075-efbf565b392b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-vprf5\" (UID: \"3eaf0419-6fce-4618-8075-efbf565b392b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:19.644513 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:19.644466    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"
Sep 17 18:40:19.920660 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:19.920629    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5"]
Sep 17 18:40:20.203156 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:20.203076    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5" event={"ID":"3eaf0419-6fce-4618-8075-efbf565b392b","Type":"ContainerStarted","Data":"5a18e59ea3abab61dd2256bf3be8d6353b68c1347bb1c944a74742e8108eef7d"}
Sep 17 18:40:22.212952 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:22.212917    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5" event={"ID":"3eaf0419-6fce-4618-8075-efbf565b392b","Type":"ContainerStarted","Data":"4ba81f34e9fd2ffda85e849f9397ee135c6b3baef412043eb3b3229bc4a7f60d"}
Sep 17 18:40:22.213363 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:22.212968    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5" event={"ID":"3eaf0419-6fce-4618-8075-efbf565b392b","Type":"ContainerStarted","Data":"285d2f05079a8e769d57e74d637a3aa9497eea25842bb6d02d33ece154816954"}
Sep 17 18:40:22.236352 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:22.236300    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-vprf5" podStartSLOduration=17.548195357 podStartE2EDuration="19.236286317s" totalImagesPullingTime="1.68809096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:22.235473679 +0000 UTC m=+95.590732477" watchObservedRunningTime="2026-09-17 18:40:22.236286317 +0000 UTC m=+95.591545089"
Sep 17 18:40:24.097525 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:24.097474    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:24.097916 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:24.097549    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:24.097916 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:24.097656    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:24.097916 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:40:24.097726    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls podName:2e6d3062-49a3-4f42-9ace-8ba0d7c90f16 nodeName:}" failed. No retries permitted until 2026-09-17 18:40:40.097708163 +0000 UTC m=+113.452966923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cbb2x" (UID: "2e6d3062-49a3-4f42-9ace-8ba0d7c90f16") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:40:24.098029 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:24.098010    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:26.113600 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.113561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:40:26.113937 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.113626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:40:26.115855 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.115837    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4494cc42-c46b-47e9-a716-42b1777a1e75-metrics-tls\") pod \"dns-default-p4xzv\" (UID: \"4494cc42-c46b-47e9-a716-42b1777a1e75\") " pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:40:26.115986 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.115966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"image-registry-7f4d48d475-4rxt2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") " pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:40:26.130936 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.130915    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6grnc\""
Sep 17 18:40:26.138607 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.138591    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:40:26.166448 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.166424    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9x8j\""
Sep 17 18:40:26.174800 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.174757    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:40:26.418057 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.417979    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4d48d475-4rxt2"]
Sep 17 18:40:26.420293 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:40:26.420264    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6124ed7a_8ab2_4762_a1fd_a676936651b2.slice/crio-f6cfa20d46956b05b436c53b525f48a55b64848e9b0f63a14af27873f7f76974 WatchSource:0}: Error finding container f6cfa20d46956b05b436c53b525f48a55b64848e9b0f63a14af27873f7f76974: Status 404 returned error can't find the container with id f6cfa20d46956b05b436c53b525f48a55b64848e9b0f63a14af27873f7f76974
Sep 17 18:40:26.456010 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.455990    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p4xzv"]
Sep 17 18:40:26.830863 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:26.830830    2557 scope.go:117] "RemoveContainer" containerID="a507649592fe78b4a1cd75e2cc8a39c60e43cee0a9fc042c944ccc96a410b7ad"
Sep 17 18:40:27.231163 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.231071    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2" event={"ID":"6124ed7a-8ab2-4762-a1fd-a676936651b2","Type":"ContainerStarted","Data":"1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf"}
Sep 17 18:40:27.231163 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.231129    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2" event={"ID":"6124ed7a-8ab2-4762-a1fd-a676936651b2","Type":"ContainerStarted","Data":"f6cfa20d46956b05b436c53b525f48a55b64848e9b0f63a14af27873f7f76974"}
Sep 17 18:40:27.231684 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.231203    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:40:27.232709 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.232675    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4xzv" event={"ID":"4494cc42-c46b-47e9-a716-42b1777a1e75","Type":"ContainerStarted","Data":"024abd8ea96983edb4db8959c630ceded046a4994b85a49656b6ee3b2ea9aa72"}
Sep 17 18:40:27.234849 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.234829    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:40:27.234957 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.234869    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j" event={"ID":"c74d2858-9009-45c9-b7c5-e2ade2bb6311","Type":"ContainerStarted","Data":"49c9ff4355672439622363c70f0ac958921ebafc1b03884795120c4ccac01565"}
Sep 17 18:40:27.235189 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.235157    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:27.256289 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.256238    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2" podStartSLOduration=98.256206119 podStartE2EDuration="1m38.256206119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:27.255619087 +0000 UTC m=+100.610877882" watchObservedRunningTime="2026-09-17 18:40:27.256206119 +0000 UTC m=+100.611464906"
Sep 17 18:40:27.278945 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.278890    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j" podStartSLOduration=21.831628078 podStartE2EDuration="24.278871539s" totalImagesPullingTime="2.447243461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:27.277867441 +0000 UTC m=+100.633126250" watchObservedRunningTime="2026-09-17 18:40:27.278871539 +0000 UTC m=+100.634130314"
Sep 17 18:40:27.833674 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:27.833643    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-pnx6j"
Sep 17 18:40:29.244599 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:29.244564    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4xzv" event={"ID":"4494cc42-c46b-47e9-a716-42b1777a1e75","Type":"ContainerStarted","Data":"c9ad04647aeaa81f13f4ec133c639a52bb47c8db66693df03dcc8bab71ae4555"}
Sep 17 18:40:29.245070 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:29.244650    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4xzv" event={"ID":"4494cc42-c46b-47e9-a716-42b1777a1e75","Type":"ContainerStarted","Data":"b07f073f37c833850dd5b7a626d7646ff2b7e56f8f2632a0a4437ee880392ebd"}
Sep 17 18:40:29.274683 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:29.274631    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p4xzv" podStartSLOduration=65.42719283 podStartE2EDuration="1m7.274615512s" totalImagesPullingTime="1.847422682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:29.26488785 +0000 UTC m=+102.620146642" watchObservedRunningTime="2026-09-17 18:40:29.274615512 +0000 UTC m=+102.629874285"
Sep 17 18:40:30.248147 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:30.248117    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:40:31.132287 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:31.132257    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x7wf6"
Sep 17 18:40:32.721956 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.721927    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-8724f"]
Sep 17 18:40:32.752119 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.752094    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"]
Sep 17 18:40:32.752279 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.752229    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f"
Sep 17 18:40:32.754965 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.754944    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-j7fgv\""
Sep 17 18:40:32.755281 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.755260    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 18:40:32.755395 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.755376    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 18:40:32.776035 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.776014    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-8724f"]
Sep 17 18:40:32.776161 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.776049    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"]
Sep 17 18:40:32.776161 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.776151    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.780708 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.780685    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:40:32.780708 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.780703    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:40:32.780883 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.780715    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 18:40:32.780883 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.780743    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 18:40:32.780987 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.780920    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:40:32.781170 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.781153    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 18:40:32.781879 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.781861    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 18:40:32.873098 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.873049    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49efff88-39af-448c-b06e-4d773e728dc6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8724f\" (UID: \"49efff88-39af-448c-b06e-4d773e728dc6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f"
Sep 17 18:40:32.873302 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.873120    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/49efff88-39af-448c-b06e-4d773e728dc6-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-8724f\" (UID: \"49efff88-39af-448c-b06e-4d773e728dc6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f"
Sep 17 18:40:32.873302 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.873180    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzd72\" (UniqueName: \"kubernetes.io/projected/8d8fa8b5-cd60-464f-9331-05c1439086e0-kube-api-access-nzd72\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.873302 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.873255    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-hub\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.873302 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.873289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8fa8b5-cd60-464f-9331-05c1439086e0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.873502 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.873340    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.873502 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.873388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.873502 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.873433    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-ca\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.974044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.973955    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.974044 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.974010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-ca\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.974278 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.974048    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49efff88-39af-448c-b06e-4d773e728dc6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8724f\" (UID: \"49efff88-39af-448c-b06e-4d773e728dc6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f"
Sep 17 18:40:32.974278 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.974075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/49efff88-39af-448c-b06e-4d773e728dc6-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-8724f\" (UID: \"49efff88-39af-448c-b06e-4d773e728dc6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f"
Sep 17 18:40:32.974278 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.974237    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzd72\" (UniqueName: \"kubernetes.io/projected/8d8fa8b5-cd60-464f-9331-05c1439086e0-kube-api-access-nzd72\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.974431 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.974304    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-hub\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.974431 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.974333    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8fa8b5-cd60-464f-9331-05c1439086e0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.974431 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.974379    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.974834 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.974813    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/49efff88-39af-448c-b06e-4d773e728dc6-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-8724f\" (UID: \"49efff88-39af-448c-b06e-4d773e728dc6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f"
Sep 17 18:40:32.975150 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.975120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8fa8b5-cd60-464f-9331-05c1439086e0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.976819 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.976791    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.976973 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.976951    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.977032 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.976951    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49efff88-39af-448c-b06e-4d773e728dc6-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-8724f\" (UID: \"49efff88-39af-448c-b06e-4d773e728dc6\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f"
Sep 17 18:40:32.977032 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.976992    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-ca\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.977366 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.977350    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8d8fa8b5-cd60-464f-9331-05c1439086e0-hub\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:32.995014 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:32.994991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzd72\" (UniqueName: \"kubernetes.io/projected/8d8fa8b5-cd60-464f-9331-05c1439086e0-kube-api-access-nzd72\") pod \"cluster-proxy-proxy-agent-685f986b99-ch8bs\" (UID: \"8d8fa8b5-cd60-464f-9331-05c1439086e0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:33.061174 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:33.061140    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f"
Sep 17 18:40:33.098050 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:33.098022    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"
Sep 17 18:40:33.351951 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:33.351922    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-8724f"]
Sep 17 18:40:33.355031 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:40:33.355003    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49efff88_39af_448c_b06e_4d773e728dc6.slice/crio-8b16416b4b18132f1278fae6e57f6d829f4b7f88b8d94efe59318a390f1d43b2 WatchSource:0}: Error finding container 8b16416b4b18132f1278fae6e57f6d829f4b7f88b8d94efe59318a390f1d43b2: Status 404 returned error can't find the container with id 8b16416b4b18132f1278fae6e57f6d829f4b7f88b8d94efe59318a390f1d43b2
Sep 17 18:40:33.399358 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:33.399329    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs"]
Sep 17 18:40:33.402459 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:40:33.402434    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d8fa8b5_cd60_464f_9331_05c1439086e0.slice/crio-a6f7d8c74298e2ec14bcf544a8529187f6ee5be86079fbb1d9f7dab861e663b7 WatchSource:0}: Error finding container a6f7d8c74298e2ec14bcf544a8529187f6ee5be86079fbb1d9f7dab861e663b7: Status 404 returned error can't find the container with id a6f7d8c74298e2ec14bcf544a8529187f6ee5be86079fbb1d9f7dab861e663b7
Sep 17 18:40:34.264673 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:34.264638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f" event={"ID":"49efff88-39af-448c-b06e-4d773e728dc6","Type":"ContainerStarted","Data":"8b16416b4b18132f1278fae6e57f6d829f4b7f88b8d94efe59318a390f1d43b2"}
Sep 17 18:40:34.265948 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:34.265902    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs" event={"ID":"8d8fa8b5-cd60-464f-9331-05c1439086e0","Type":"ContainerStarted","Data":"a6f7d8c74298e2ec14bcf544a8529187f6ee5be86079fbb1d9f7dab861e663b7"}
Sep 17 18:40:35.271019 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:35.270984    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f" event={"ID":"49efff88-39af-448c-b06e-4d773e728dc6","Type":"ContainerStarted","Data":"9e246c855f7cab9bd2ee309d5f64d9f9c317ac9dcdd67bc6af93c0afeab72c56"}
Sep 17 18:40:35.302476 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:35.302424    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-8724f" podStartSLOduration=1.9457319819999999 podStartE2EDuration="3.302402626s" totalImagesPullingTime="1.356670644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:35.297076776 +0000 UTC m=+108.652335568" watchObservedRunningTime="2026-09-17 18:40:35.302402626 +0000 UTC m=+108.657661400"
Sep 17 18:40:40.133729 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:40.133684    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:40.136324 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:40.136297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e6d3062-49a3-4f42-9ace-8ba0d7c90f16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cbb2x\" (UID: \"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16\") " pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:40.253567 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:40.253537    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p4xzv"
Sep 17 18:40:40.420887 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:40.420807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cbb2x"
Sep 17 18:40:40.943085 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:40.942994    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cbb2x"]
Sep 17 18:40:41.085390 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:40:41.085357    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6d3062_49a3_4f42_9ace_8ba0d7c90f16.slice/crio-399977e01a9850d774f0d5adb634eee8976c39f780ff7b3ec4a11b76f518c64c WatchSource:0}: Error finding container 399977e01a9850d774f0d5adb634eee8976c39f780ff7b3ec4a11b76f518c64c: Status 404 returned error can't find the container with id 399977e01a9850d774f0d5adb634eee8976c39f780ff7b3ec4a11b76f518c64c
Sep 17 18:40:41.296190 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:41.296156    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cbb2x" event={"ID":"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16","Type":"ContainerStarted","Data":"3ca9400393d775798c95609feabd4ffa1fbde7f41bfc25b0cd1fa434cd338924"}
Sep 17 18:40:41.296622 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:41.296237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cbb2x" event={"ID":"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16","Type":"ContainerStarted","Data":"399977e01a9850d774f0d5adb634eee8976c39f780ff7b3ec4a11b76f518c64c"}
Sep 17 18:40:41.298109 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:41.298087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs" event={"ID":"8d8fa8b5-cd60-464f-9331-05c1439086e0","Type":"ContainerStarted","Data":"83d3c1778404e3ba49f5dcd391a27520d3cf152df29761a1f356ae43a60d3fe6"}
Sep 17 18:40:41.298199 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:41.298122    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs" event={"ID":"8d8fa8b5-cd60-464f-9331-05c1439086e0","Type":"ContainerStarted","Data":"d49be66c36114f70c71483b40f294690bdaf4f50b701f0b48419cfcfa9900149"}
Sep 17 18:40:41.298199 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:41.298142    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs" event={"ID":"8d8fa8b5-cd60-464f-9331-05c1439086e0","Type":"ContainerStarted","Data":"0d3260c12944c16e1b0a574c5e844c38c725c9318368a62dcfb4faad3b71627e"}
Sep 17 18:40:41.319127 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:41.319076    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-685f986b99-ch8bs" podStartSLOduration=1.619742062 podStartE2EDuration="9.319058525s" totalImagesPullingTime="7.699316463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:41.318301174 +0000 UTC m=+114.673559952" watchObservedRunningTime="2026-09-17 18:40:41.319058525 +0000 UTC m=+114.674317344"
Sep 17 18:40:42.308158 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:42.308117    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cbb2x" event={"ID":"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16","Type":"ContainerStarted","Data":"144feb4406d80ba6b96300b6b10da94f86c486cfff7a04905783a0dc45573fdc"}
Sep 17 18:40:43.312864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:43.312780    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cbb2x" event={"ID":"2e6d3062-49a3-4f42-9ace-8ba0d7c90f16","Type":"ContainerStarted","Data":"f91a70b625af81637d1b7673c5b0d3846ee75b39955aaceb48d51d6db70559a9"}
Sep 17 18:40:43.334064 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:43.334014    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cbb2x" podStartSLOduration=33.562677929 podStartE2EDuration="35.333999239s" totalImagesPullingTime="1.77132131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:43.33206056 +0000 UTC m=+116.687319393" watchObservedRunningTime="2026-09-17 18:40:43.333999239 +0000 UTC m=+116.689258056"
Sep 17 18:40:46.143324 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:46.143281    2557 patch_prober.go:28] interesting pod/image-registry-7f4d48d475-4rxt2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 18:40:46.143699 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:46.143337    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2" podUID="6124ed7a-8ab2-4762-a1fd-a676936651b2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:47.907354 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:47.907321    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lkpgf"]
Sep 17 18:40:47.910589 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:47.910574    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:47.914533 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:47.914515    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:40:47.915045 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:47.915028    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:40:47.915670 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:47.915647    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:40:47.915771 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:47.915678    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l9q9n\""
Sep 17 18:40:47.916014 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:47.916000    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:40:47.916131 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:47.916109    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:40:47.916183 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:47.916136    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:40:48.093198 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.093168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-textfile\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.093198 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.093202    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-wtmp\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.093397 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.093247    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01856e90-5b5e-4eae-a91d-52466f206b12-root\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.093397 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.093270    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01856e90-5b5e-4eae-a91d-52466f206b12-sys\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.093397 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.093287    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.093397 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.093334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7fv\" (UniqueName: \"kubernetes.io/projected/01856e90-5b5e-4eae-a91d-52466f206b12-kube-api-access-vq7fv\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.093397 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.093360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-tls\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.093550 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.093405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-accelerators-collector-config\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.093550 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.093423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01856e90-5b5e-4eae-a91d-52466f206b12-metrics-client-ca\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.193980 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.193885    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7fv\" (UniqueName: \"kubernetes.io/projected/01856e90-5b5e-4eae-a91d-52466f206b12-kube-api-access-vq7fv\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.193980 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.193925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-tls\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.193980 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.193970    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-accelerators-collector-config\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194307 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.193993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01856e90-5b5e-4eae-a91d-52466f206b12-metrics-client-ca\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194307 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-textfile\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194307 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194091    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-wtmp\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194307 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194125    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01856e90-5b5e-4eae-a91d-52466f206b12-root\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194307 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01856e90-5b5e-4eae-a91d-52466f206b12-sys\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194307 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194307 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194254    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01856e90-5b5e-4eae-a91d-52466f206b12-root\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194645 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01856e90-5b5e-4eae-a91d-52466f206b12-sys\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194645 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194312    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-wtmp\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194645 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194535    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-textfile\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194850 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194809    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-accelerators-collector-config\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.194970 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.194892    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01856e90-5b5e-4eae-a91d-52466f206b12-metrics-client-ca\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.196318 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.196297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-tls\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.196446 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.196429    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01856e90-5b5e-4eae-a91d-52466f206b12-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.205462 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.205443    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7fv\" (UniqueName: \"kubernetes.io/projected/01856e90-5b5e-4eae-a91d-52466f206b12-kube-api-access-vq7fv\") pod \"node-exporter-lkpgf\" (UID: \"01856e90-5b5e-4eae-a91d-52466f206b12\") " pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.219774 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.219750    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lkpgf"
Sep 17 18:40:48.231893 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:40:48.231870    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01856e90_5b5e_4eae_a91d_52466f206b12.slice/crio-cc2dcb63b53fa3b1237410a93563d22651860bfc058f131fbfed24a8a08fbdd9 WatchSource:0}: Error finding container cc2dcb63b53fa3b1237410a93563d22651860bfc058f131fbfed24a8a08fbdd9: Status 404 returned error can't find the container with id cc2dcb63b53fa3b1237410a93563d22651860bfc058f131fbfed24a8a08fbdd9
Sep 17 18:40:48.244143 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.244124    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:40:48.330043 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:48.330016    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lkpgf" event={"ID":"01856e90-5b5e-4eae-a91d-52466f206b12","Type":"ContainerStarted","Data":"cc2dcb63b53fa3b1237410a93563d22651860bfc058f131fbfed24a8a08fbdd9"}
Sep 17 18:40:49.335925 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:49.335884    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lkpgf" event={"ID":"01856e90-5b5e-4eae-a91d-52466f206b12","Type":"ContainerStarted","Data":"81237d069a9785adb4a833eb01de5732de620512234ee56563d9d244b7b1e0ae"}
Sep 17 18:40:50.340456 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:50.340417    2557 generic.go:356] "Generic (PLEG): container finished" podID="01856e90-5b5e-4eae-a91d-52466f206b12" containerID="81237d069a9785adb4a833eb01de5732de620512234ee56563d9d244b7b1e0ae" exitCode=0
Sep 17 18:40:50.340818 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:50.340465    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lkpgf" event={"ID":"01856e90-5b5e-4eae-a91d-52466f206b12","Type":"ContainerDied","Data":"81237d069a9785adb4a833eb01de5732de620512234ee56563d9d244b7b1e0ae"}
Sep 17 18:40:51.346371 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:51.346333    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lkpgf" event={"ID":"01856e90-5b5e-4eae-a91d-52466f206b12","Type":"ContainerStarted","Data":"310d7b18aeafbecf202f0545a5c8259cfce72a07d261de726a91719c9e95a070"}
Sep 17 18:40:51.346798 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:51.346400    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lkpgf" event={"ID":"01856e90-5b5e-4eae-a91d-52466f206b12","Type":"ContainerStarted","Data":"aa459bd5bf85cc4e5ad7b0b6c75bffd644864d934bc39201136a112d6e570441"}
Sep 17 18:40:51.377550 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:51.377504    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lkpgf" podStartSLOduration=3.326856196 podStartE2EDuration="4.377487606s" totalImagesPullingTime="988.419467ms" totalInitContainerRuntime="62.211943ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:51.375790129 +0000 UTC m=+124.731048921" watchObservedRunningTime="2026-09-17 18:40:51.377487606 +0000 UTC m=+124.732746378"
Sep 17 18:40:54.717096 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:54.717059    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4d48d475-4rxt2"]
Sep 17 18:40:58.671401 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:58.671361    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:40:58.673606 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:58.673589    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f20d3b-b7f6-435b-9375-bef4013e5de5-metrics-certs\") pod \"network-metrics-daemon-qvr8b\" (UID: \"a0f20d3b-b7f6-435b-9375-bef4013e5de5\") " pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:40:58.944551 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:58.944474    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nghzc\""
Sep 17 18:40:58.952947 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:58.952917    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvr8b"
Sep 17 18:40:59.225094 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:59.225037    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvr8b"]
Sep 17 18:40:59.227350 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:40:59.227324    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f20d3b_b7f6_435b_9375_bef4013e5de5.slice/crio-0113fdfeb53a887a4e0b79deddf6a62301a3b224f49e9564167705b99f220296 WatchSource:0}: Error finding container 0113fdfeb53a887a4e0b79deddf6a62301a3b224f49e9564167705b99f220296: Status 404 returned error can't find the container with id 0113fdfeb53a887a4e0b79deddf6a62301a3b224f49e9564167705b99f220296
Sep 17 18:40:59.373126 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:40:59.373088    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvr8b" event={"ID":"a0f20d3b-b7f6-435b-9375-bef4013e5de5","Type":"ContainerStarted","Data":"0113fdfeb53a887a4e0b79deddf6a62301a3b224f49e9564167705b99f220296"}
Sep 17 18:41:00.377620 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:00.377592    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvr8b" event={"ID":"a0f20d3b-b7f6-435b-9375-bef4013e5de5","Type":"ContainerStarted","Data":"12c49549d12b22346b232d204611bb5fd6a8de1f33ac8eb97c69841bbb5213d7"}
Sep 17 18:41:01.382844 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:01.382806    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvr8b" event={"ID":"a0f20d3b-b7f6-435b-9375-bef4013e5de5","Type":"ContainerStarted","Data":"56cdeb58fda182cc54df4bed84d042c8699cd79cd1d77426923224345155b56b"}
Sep 17 18:41:01.412411 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:01.412361    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qvr8b" podStartSLOduration=131.42772141 podStartE2EDuration="2m12.412337565s" totalImagesPullingTime="984.616155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:01.410149033 +0000 UTC m=+134.765407819" watchObservedRunningTime="2026-09-17 18:41:01.412337565 +0000 UTC m=+134.767596385"
Sep 17 18:41:19.737812 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:19.737771    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2" podUID="6124ed7a-8ab2-4762-a1fd-a676936651b2" containerName="registry" containerID="cri-o://1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf" gracePeriod=30
Sep 17 18:41:19.980720 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:19.980698    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:41:20.023774 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.023703    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6124ed7a-8ab2-4762-a1fd-a676936651b2-ca-trust-extracted\") pod \"6124ed7a-8ab2-4762-a1fd-a676936651b2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") "
Sep 17 18:41:20.023774 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.023744    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-certificates\") pod \"6124ed7a-8ab2-4762-a1fd-a676936651b2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") "
Sep 17 18:41:20.023987 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.023790    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqgkk\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-kube-api-access-sqgkk\") pod \"6124ed7a-8ab2-4762-a1fd-a676936651b2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") "
Sep 17 18:41:20.023987 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.023827    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") pod \"6124ed7a-8ab2-4762-a1fd-a676936651b2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") "
Sep 17 18:41:20.023987 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.023869    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-image-registry-private-configuration\") pod \"6124ed7a-8ab2-4762-a1fd-a676936651b2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") "
Sep 17 18:41:20.024133 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.024004    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-bound-sa-token\") pod \"6124ed7a-8ab2-4762-a1fd-a676936651b2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") "
Sep 17 18:41:20.024133 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.024056    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-trusted-ca\") pod \"6124ed7a-8ab2-4762-a1fd-a676936651b2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") "
Sep 17 18:41:20.024133 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.024115    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-installation-pull-secrets\") pod \"6124ed7a-8ab2-4762-a1fd-a676936651b2\" (UID: \"6124ed7a-8ab2-4762-a1fd-a676936651b2\") "
Sep 17 18:41:20.024318 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.024256    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6124ed7a-8ab2-4762-a1fd-a676936651b2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:20.024427 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.024408    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-certificates\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:41:20.024733 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.024703    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6124ed7a-8ab2-4762-a1fd-a676936651b2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:20.026417 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.026394    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6124ed7a-8ab2-4762-a1fd-a676936651b2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:20.026613 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.026581    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6124ed7a-8ab2-4762-a1fd-a676936651b2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:20.026723 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.026609    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-kube-api-access-sqgkk" (OuterVolumeSpecName: "kube-api-access-sqgkk") pod "6124ed7a-8ab2-4762-a1fd-a676936651b2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2"). InnerVolumeSpecName "kube-api-access-sqgkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:20.026775 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.026728    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6124ed7a-8ab2-4762-a1fd-a676936651b2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:20.026813 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.026790    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6124ed7a-8ab2-4762-a1fd-a676936651b2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:20.034844 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.034819    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6124ed7a-8ab2-4762-a1fd-a676936651b2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6124ed7a-8ab2-4762-a1fd-a676936651b2" (UID: "6124ed7a-8ab2-4762-a1fd-a676936651b2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:20.124971 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.124932    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6124ed7a-8ab2-4762-a1fd-a676936651b2-trusted-ca\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:41:20.124971 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.124963    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-installation-pull-secrets\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:41:20.124971 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.124974    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6124ed7a-8ab2-4762-a1fd-a676936651b2-ca-trust-extracted\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:41:20.124971 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.124982    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqgkk\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-kube-api-access-sqgkk\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:41:20.125251 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.124990    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-registry-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:41:20.125251 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.124998    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6124ed7a-8ab2-4762-a1fd-a676936651b2-image-registry-private-configuration\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:41:20.125251 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.125007    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6124ed7a-8ab2-4762-a1fd-a676936651b2-bound-sa-token\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:41:20.454009 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.453980    2557 generic.go:356] "Generic (PLEG): container finished" podID="6124ed7a-8ab2-4762-a1fd-a676936651b2" containerID="1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf" exitCode=0
Sep 17 18:41:20.454152 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.454014    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2" event={"ID":"6124ed7a-8ab2-4762-a1fd-a676936651b2","Type":"ContainerDied","Data":"1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf"}
Sep 17 18:41:20.454152 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.454066    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2" event={"ID":"6124ed7a-8ab2-4762-a1fd-a676936651b2","Type":"ContainerDied","Data":"f6cfa20d46956b05b436c53b525f48a55b64848e9b0f63a14af27873f7f76974"}
Sep 17 18:41:20.454152 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.454084    2557 scope.go:117] "RemoveContainer" containerID="1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf"
Sep 17 18:41:20.454152 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.454025    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4d48d475-4rxt2"
Sep 17 18:41:20.462864 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.462844    2557 scope.go:117] "RemoveContainer" containerID="1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf"
Sep 17 18:41:20.463123 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:41:20.463104    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf\": container with ID starting with 1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf not found: ID does not exist" containerID="1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf"
Sep 17 18:41:20.463171 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.463130    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf"} err="failed to get container status \"1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf\": rpc error: code = NotFound desc = could not find container \"1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf\": container with ID starting with 1bc57936f6bffd6cff4cfa9a1f824a5b1ef50a4d1291cefd640e99c30cc336cf not found: ID does not exist"
Sep 17 18:41:20.481605 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.481575    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f4d48d475-4rxt2"]
Sep 17 18:41:20.484790 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.484770    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f4d48d475-4rxt2"]
Sep 17 18:41:20.835709 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:20.835673    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6124ed7a-8ab2-4762-a1fd-a676936651b2" path="/var/lib/kubelet/pods/6124ed7a-8ab2-4762-a1fd-a676936651b2/volumes"
Sep 17 18:41:38.515846 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:38.515810    2557 generic.go:356] "Generic (PLEG): container finished" podID="5f273124-4276-4588-962a-4424f23b76b8" containerID="ed67c71c9815da56a5d4bae5e3f4c6c5ec8b20959fe4e125067428eb9dc50b2a" exitCode=0
Sep 17 18:41:38.516262 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:38.515881    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk" event={"ID":"5f273124-4276-4588-962a-4424f23b76b8","Type":"ContainerDied","Data":"ed67c71c9815da56a5d4bae5e3f4c6c5ec8b20959fe4e125067428eb9dc50b2a"}
Sep 17 18:41:38.516262 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:38.516174    2557 scope.go:117] "RemoveContainer" containerID="ed67c71c9815da56a5d4bae5e3f4c6c5ec8b20959fe4e125067428eb9dc50b2a"
Sep 17 18:41:39.521096 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:41:39.521063    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-vv6wk" event={"ID":"5f273124-4276-4588-962a-4424f23b76b8","Type":"ContainerStarted","Data":"023f59e3b7ff80d07c1a8f415ab53051a95fcbe540e460ab92ee9b082e7d2b6c"}
Sep 17 18:43:48.688790 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:43:48.688763    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:43:48.692145 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:43:48.692124    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:43:48.692533 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:43:48.692517    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:43:48.695577 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:43:48.695557    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:43:48.699252 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:43:48.699230    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:46:52.974025 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:52.973988    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"]
Sep 17 18:46:52.974511 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:52.974246    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6124ed7a-8ab2-4762-a1fd-a676936651b2" containerName="registry"
Sep 17 18:46:52.974511 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:52.974258    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6124ed7a-8ab2-4762-a1fd-a676936651b2" containerName="registry"
Sep 17 18:46:52.974511 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:52.974300    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6124ed7a-8ab2-4762-a1fd-a676936651b2" containerName="registry"
Sep 17 18:46:52.977126 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:52.977110    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:52.981809 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:52.981789    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-94jn6\""
Sep 17 18:46:52.983415 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:52.983039    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:46:52.983415 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:52.983290    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:46:52.999691 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:52.999662    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"]
Sep 17 18:46:53.098617 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.098580    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.098803 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.098638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.098803 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.098686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvfw\" (UniqueName: \"kubernetes.io/projected/edfb5ef7-6564-4316-bf56-5458fd6b0224-kube-api-access-rrvfw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.199184 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.199147    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.199360 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.199197    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvfw\" (UniqueName: \"kubernetes.io/projected/edfb5ef7-6564-4316-bf56-5458fd6b0224-kube-api-access-rrvfw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.199360 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.199253    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.199584 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.199565    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.199643 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.199624    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.211972 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.211949    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvfw\" (UniqueName: \"kubernetes.io/projected/edfb5ef7-6564-4316-bf56-5458fd6b0224-kube-api-access-rrvfw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.285991 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.285912    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:46:53.585853 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.585824    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"]
Sep 17 18:46:53.587679 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:46:53.587650    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfb5ef7_6564_4316_bf56_5458fd6b0224.slice/crio-ac55bb98aa576549e7e422d7212c9e54d671f174e1738c0427a6f9af65dfa208 WatchSource:0}: Error finding container ac55bb98aa576549e7e422d7212c9e54d671f174e1738c0427a6f9af65dfa208: Status 404 returned error can't find the container with id ac55bb98aa576549e7e422d7212c9e54d671f174e1738c0427a6f9af65dfa208
Sep 17 18:46:53.591767 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:53.591747    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:46:54.590973 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:54.590938    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9" event={"ID":"edfb5ef7-6564-4316-bf56-5458fd6b0224","Type":"ContainerStarted","Data":"ac55bb98aa576549e7e422d7212c9e54d671f174e1738c0427a6f9af65dfa208"}
Sep 17 18:46:59.610474 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:59.610440    2557 generic.go:356] "Generic (PLEG): container finished" podID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerID="025127140e4ff9e0f7aca3a2a98f133fd4a2f92dd5c65d9c8c9fe80627a40210" exitCode=0
Sep 17 18:46:59.610976 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:46:59.610501    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9" event={"ID":"edfb5ef7-6564-4316-bf56-5458fd6b0224","Type":"ContainerDied","Data":"025127140e4ff9e0f7aca3a2a98f133fd4a2f92dd5c65d9c8c9fe80627a40210"}
Sep 17 18:47:01.620664 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:01.620630    2557 generic.go:356] "Generic (PLEG): container finished" podID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerID="f4840ee5c77c077bc72bfb887e9fdaddc20221030ee05badc9ba8ed3c91d8478" exitCode=0
Sep 17 18:47:01.621041 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:01.620703    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9" event={"ID":"edfb5ef7-6564-4316-bf56-5458fd6b0224","Type":"ContainerDied","Data":"f4840ee5c77c077bc72bfb887e9fdaddc20221030ee05badc9ba8ed3c91d8478"}
Sep 17 18:47:11.665526 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:11.665501    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9" event={"ID":"edfb5ef7-6564-4316-bf56-5458fd6b0224","Type":"ContainerStarted","Data":"bf081318b49729ddfe82d807a3d9b6699c8fe030e83aeb91cdb829db323ac855"}
Sep 17 18:47:12.670888 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:12.670855    2557 generic.go:356] "Generic (PLEG): container finished" podID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerID="bf081318b49729ddfe82d807a3d9b6699c8fe030e83aeb91cdb829db323ac855" exitCode=0
Sep 17 18:47:12.671288 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:12.670920    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9" event={"ID":"edfb5ef7-6564-4316-bf56-5458fd6b0224","Type":"ContainerDied","Data":"bf081318b49729ddfe82d807a3d9b6699c8fe030e83aeb91cdb829db323ac855"}
Sep 17 18:47:13.806626 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:13.806602    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:47:13.958495 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:13.958401    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrvfw\" (UniqueName: \"kubernetes.io/projected/edfb5ef7-6564-4316-bf56-5458fd6b0224-kube-api-access-rrvfw\") pod \"edfb5ef7-6564-4316-bf56-5458fd6b0224\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") "
Sep 17 18:47:13.958495 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:13.958461    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-util\") pod \"edfb5ef7-6564-4316-bf56-5458fd6b0224\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") "
Sep 17 18:47:13.958495 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:13.958485    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-bundle\") pod \"edfb5ef7-6564-4316-bf56-5458fd6b0224\" (UID: \"edfb5ef7-6564-4316-bf56-5458fd6b0224\") "
Sep 17 18:47:13.959112 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:13.959075    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-bundle" (OuterVolumeSpecName: "bundle") pod "edfb5ef7-6564-4316-bf56-5458fd6b0224" (UID: "edfb5ef7-6564-4316-bf56-5458fd6b0224"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:13.960787 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:13.960752    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edfb5ef7-6564-4316-bf56-5458fd6b0224-kube-api-access-rrvfw" (OuterVolumeSpecName: "kube-api-access-rrvfw") pod "edfb5ef7-6564-4316-bf56-5458fd6b0224" (UID: "edfb5ef7-6564-4316-bf56-5458fd6b0224"). InnerVolumeSpecName "kube-api-access-rrvfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:13.962934 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:13.962915    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-util" (OuterVolumeSpecName: "util") pod "edfb5ef7-6564-4316-bf56-5458fd6b0224" (UID: "edfb5ef7-6564-4316-bf56-5458fd6b0224"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:14.059887 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:14.059852    2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-util\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:47:14.059887 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:14.059880    2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/edfb5ef7-6564-4316-bf56-5458fd6b0224-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:47:14.059887 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:14.059890    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrvfw\" (UniqueName: \"kubernetes.io/projected/edfb5ef7-6564-4316-bf56-5458fd6b0224-kube-api-access-rrvfw\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:47:14.679949 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:14.679909    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9" event={"ID":"edfb5ef7-6564-4316-bf56-5458fd6b0224","Type":"ContainerDied","Data":"ac55bb98aa576549e7e422d7212c9e54d671f174e1738c0427a6f9af65dfa208"}
Sep 17 18:47:14.680114 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:14.679956    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ag8qk9"
Sep 17 18:47:14.680114 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:14.679961    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac55bb98aa576549e7e422d7212c9e54d671f174e1738c0427a6f9af65dfa208"
Sep 17 18:47:26.419307 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.419258    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-jtzqw"]
Sep 17 18:47:26.419782 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.419505    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerName="extract"
Sep 17 18:47:26.419782 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.419516    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerName="extract"
Sep 17 18:47:26.419782 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.419532    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerName="pull"
Sep 17 18:47:26.419782 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.419538    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerName="pull"
Sep 17 18:47:26.419782 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.419546    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerName="util"
Sep 17 18:47:26.419782 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.419551    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerName="util"
Sep 17 18:47:26.419782 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.419593    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="edfb5ef7-6564-4316-bf56-5458fd6b0224" containerName="extract"
Sep 17 18:47:26.422628 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.422611    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:26.427510 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.427489    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:47:26.428625 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.428606    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:47:26.429251 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.429232    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 18:47:26.429477 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.429459    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xmlsk\""
Sep 17 18:47:26.429574 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.429525    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:47:26.430003 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.429987    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 18:47:26.436788 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.436760    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-jtzqw"]
Sep 17 18:47:26.450710 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.450686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cd3ff771-d54a-4ea0-b856-518c605640dd-cabundle0\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:26.450810 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.450728    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp757\" (UniqueName: \"kubernetes.io/projected/cd3ff771-d54a-4ea0-b856-518c605640dd-kube-api-access-rp757\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:26.450810 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.450766    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd3ff771-d54a-4ea0-b856-518c605640dd-certificates\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:26.551069 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.551031    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cd3ff771-d54a-4ea0-b856-518c605640dd-cabundle0\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:26.551271 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.551079    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp757\" (UniqueName: \"kubernetes.io/projected/cd3ff771-d54a-4ea0-b856-518c605640dd-kube-api-access-rp757\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:26.551271 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.551105    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd3ff771-d54a-4ea0-b856-518c605640dd-certificates\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:26.551271 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:47:26.551204    2557 secret.go:281] references non-existent secret key: ca.crt
Sep 17 18:47:26.551271 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:47:26.551236    2557 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 18:47:26.551271 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:47:26.551245    2557 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-jtzqw: references non-existent secret key: ca.crt
Sep 17 18:47:26.551454 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:47:26.551297    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd3ff771-d54a-4ea0-b856-518c605640dd-certificates podName:cd3ff771-d54a-4ea0-b856-518c605640dd nodeName:}" failed. No retries permitted until 2026-09-17 18:47:27.051279232 +0000 UTC m=+520.406537989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cd3ff771-d54a-4ea0-b856-518c605640dd-certificates") pod "keda-operator-579c8958f7-jtzqw" (UID: "cd3ff771-d54a-4ea0-b856-518c605640dd") : references non-existent secret key: ca.crt
Sep 17 18:47:26.551710 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.551689    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cd3ff771-d54a-4ea0-b856-518c605640dd-cabundle0\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:26.564835 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.564804    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp757\" (UniqueName: \"kubernetes.io/projected/cd3ff771-d54a-4ea0-b856-518c605640dd-kube-api-access-rp757\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:26.697765 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.697682    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"]
Sep 17 18:47:26.700736 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.700721    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.703850 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.703830    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 18:47:26.713020 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.712996    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"]
Sep 17 18:47:26.752938 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.752909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c1cd129-b6b9-4610-b232-8fcd9d78d65f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-2rhkf\" (UID: \"8c1cd129-b6b9-4610-b232-8fcd9d78d65f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.753089 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.752957    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8c1cd129-b6b9-4610-b232-8fcd9d78d65f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-2rhkf\" (UID: \"8c1cd129-b6b9-4610-b232-8fcd9d78d65f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.753089 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.752975    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v44p2\" (UniqueName: \"kubernetes.io/projected/8c1cd129-b6b9-4610-b232-8fcd9d78d65f-kube-api-access-v44p2\") pod \"keda-metrics-apiserver-77857b7b98-2rhkf\" (UID: \"8c1cd129-b6b9-4610-b232-8fcd9d78d65f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.853962 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.853931    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c1cd129-b6b9-4610-b232-8fcd9d78d65f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-2rhkf\" (UID: \"8c1cd129-b6b9-4610-b232-8fcd9d78d65f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.854114 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.853992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8c1cd129-b6b9-4610-b232-8fcd9d78d65f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-2rhkf\" (UID: \"8c1cd129-b6b9-4610-b232-8fcd9d78d65f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.854114 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.854019    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v44p2\" (UniqueName: \"kubernetes.io/projected/8c1cd129-b6b9-4610-b232-8fcd9d78d65f-kube-api-access-v44p2\") pod \"keda-metrics-apiserver-77857b7b98-2rhkf\" (UID: \"8c1cd129-b6b9-4610-b232-8fcd9d78d65f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.854361 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.854340    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8c1cd129-b6b9-4610-b232-8fcd9d78d65f-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-2rhkf\" (UID: \"8c1cd129-b6b9-4610-b232-8fcd9d78d65f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.856507 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.856488    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c1cd129-b6b9-4610-b232-8fcd9d78d65f-certificates\") pod \"keda-metrics-apiserver-77857b7b98-2rhkf\" (UID: \"8c1cd129-b6b9-4610-b232-8fcd9d78d65f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.872877 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.872843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v44p2\" (UniqueName: \"kubernetes.io/projected/8c1cd129-b6b9-4610-b232-8fcd9d78d65f-kube-api-access-v44p2\") pod \"keda-metrics-apiserver-77857b7b98-2rhkf\" (UID: \"8c1cd129-b6b9-4610-b232-8fcd9d78d65f\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:26.951570 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.951481    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-4kkhn"]
Sep 17 18:47:26.954614 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.954592    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:26.959400 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.959383    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 18:47:26.974315 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:26.974289    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-4kkhn"]
Sep 17 18:47:27.010552 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.010522    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:27.055089 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.055045    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd3ff771-d54a-4ea0-b856-518c605640dd-certificates\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:27.055279 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.055100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvc9r\" (UniqueName: \"kubernetes.io/projected/f19d1e59-d9e9-4112-ae36-3f31f35f09b5-kube-api-access-nvc9r\") pod \"keda-admission-6c4d864b86-4kkhn\" (UID: \"f19d1e59-d9e9-4112-ae36-3f31f35f09b5\") " pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:27.055279 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.055129    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f19d1e59-d9e9-4112-ae36-3f31f35f09b5-certificates\") pod \"keda-admission-6c4d864b86-4kkhn\" (UID: \"f19d1e59-d9e9-4112-ae36-3f31f35f09b5\") " pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:27.058809 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.058758    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cd3ff771-d54a-4ea0-b856-518c605640dd-certificates\") pod \"keda-operator-579c8958f7-jtzqw\" (UID: \"cd3ff771-d54a-4ea0-b856-518c605640dd\") " pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:27.156089 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.156052    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvc9r\" (UniqueName: \"kubernetes.io/projected/f19d1e59-d9e9-4112-ae36-3f31f35f09b5-kube-api-access-nvc9r\") pod \"keda-admission-6c4d864b86-4kkhn\" (UID: \"f19d1e59-d9e9-4112-ae36-3f31f35f09b5\") " pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:27.156089 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.156091    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f19d1e59-d9e9-4112-ae36-3f31f35f09b5-certificates\") pod \"keda-admission-6c4d864b86-4kkhn\" (UID: \"f19d1e59-d9e9-4112-ae36-3f31f35f09b5\") " pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:27.158472 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.158455    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f19d1e59-d9e9-4112-ae36-3f31f35f09b5-certificates\") pod \"keda-admission-6c4d864b86-4kkhn\" (UID: \"f19d1e59-d9e9-4112-ae36-3f31f35f09b5\") " pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:27.169629 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.169607    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvc9r\" (UniqueName: \"kubernetes.io/projected/f19d1e59-d9e9-4112-ae36-3f31f35f09b5-kube-api-access-nvc9r\") pod \"keda-admission-6c4d864b86-4kkhn\" (UID: \"f19d1e59-d9e9-4112-ae36-3f31f35f09b5\") " pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:27.264820 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.264757    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:27.293269 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.293241    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"]
Sep 17 18:47:27.296997 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:47:27.296971    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1cd129_b6b9_4610_b232_8fcd9d78d65f.slice/crio-1ed10671ff940bce8cf8cd2c597085865d27508b551752ea6574a80e8dcdb5fe WatchSource:0}: Error finding container 1ed10671ff940bce8cf8cd2c597085865d27508b551752ea6574a80e8dcdb5fe: Status 404 returned error can't find the container with id 1ed10671ff940bce8cf8cd2c597085865d27508b551752ea6574a80e8dcdb5fe
Sep 17 18:47:27.332520 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.332497    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:27.541198 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.541112    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-4kkhn"]
Sep 17 18:47:27.544303 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:47:27.544275    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19d1e59_d9e9_4112_ae36_3f31f35f09b5.slice/crio-515a4e8fe18b14289f084e8973d22d42dbadcd7278b29b85ff70884e1ae81866 WatchSource:0}: Error finding container 515a4e8fe18b14289f084e8973d22d42dbadcd7278b29b85ff70884e1ae81866: Status 404 returned error can't find the container with id 515a4e8fe18b14289f084e8973d22d42dbadcd7278b29b85ff70884e1ae81866
Sep 17 18:47:27.621576 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.621550    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-jtzqw"]
Sep 17 18:47:27.625463 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:47:27.625434    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3ff771_d54a_4ea0_b856_518c605640dd.slice/crio-1bc644dcd725cb6d1597e86baf947d573f9f90e93c8435f5fb9720e0d6a0f6cb WatchSource:0}: Error finding container 1bc644dcd725cb6d1597e86baf947d573f9f90e93c8435f5fb9720e0d6a0f6cb: Status 404 returned error can't find the container with id 1bc644dcd725cb6d1597e86baf947d573f9f90e93c8435f5fb9720e0d6a0f6cb
Sep 17 18:47:27.727701 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.727643    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-jtzqw" event={"ID":"cd3ff771-d54a-4ea0-b856-518c605640dd","Type":"ContainerStarted","Data":"1bc644dcd725cb6d1597e86baf947d573f9f90e93c8435f5fb9720e0d6a0f6cb"}
Sep 17 18:47:27.728938 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.728915    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-4kkhn" event={"ID":"f19d1e59-d9e9-4112-ae36-3f31f35f09b5","Type":"ContainerStarted","Data":"515a4e8fe18b14289f084e8973d22d42dbadcd7278b29b85ff70884e1ae81866"}
Sep 17 18:47:27.729970 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:27.729951    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf" event={"ID":"8c1cd129-b6b9-4610-b232-8fcd9d78d65f","Type":"ContainerStarted","Data":"1ed10671ff940bce8cf8cd2c597085865d27508b551752ea6574a80e8dcdb5fe"}
Sep 17 18:47:30.745811 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:30.745723    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-4kkhn" event={"ID":"f19d1e59-d9e9-4112-ae36-3f31f35f09b5","Type":"ContainerStarted","Data":"9064a17a68a4f1eb10ec5db001641929cdfe69c6b397607960aadc0fc4ae7d3e"}
Sep 17 18:47:30.746271 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:30.745844    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:30.788921 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:30.788782    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-4kkhn" podStartSLOduration=2.56393731 podStartE2EDuration="4.788760795s" totalImagesPullingTime="2.224823485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:30.787424141 +0000 UTC m=+524.142682938" watchObservedRunningTime="2026-09-17 18:47:30.788760795 +0000 UTC m=+524.144019573"
Sep 17 18:47:33.759644 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:33.759607    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf" event={"ID":"8c1cd129-b6b9-4610-b232-8fcd9d78d65f","Type":"ContainerStarted","Data":"0c7995dc69477cf388444b167d10f77588674475a4445166aacd19f84ec6dbb0"}
Sep 17 18:47:33.760141 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:33.759676    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:33.761290 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:33.761269    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-jtzqw" event={"ID":"cd3ff771-d54a-4ea0-b856-518c605640dd","Type":"ContainerStarted","Data":"4d237c16de47a82ca34697f00fac444ad797461584d13e29643e66dd8e3f59bc"}
Sep 17 18:47:33.761396 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:33.761384    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:47:33.789848 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:33.789803    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf" podStartSLOduration=2.159325172 podStartE2EDuration="7.789789176s" totalImagesPullingTime="5.630464004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:33.789189017 +0000 UTC m=+527.144447811" watchObservedRunningTime="2026-09-17 18:47:33.789789176 +0000 UTC m=+527.145047949"
Sep 17 18:47:33.821812 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:33.821770    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-jtzqw" podStartSLOduration=2.519727567 podStartE2EDuration="7.821756291s" totalImagesPullingTime="5.302028724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:33.821377311 +0000 UTC m=+527.176636103" watchObservedRunningTime="2026-09-17 18:47:33.821756291 +0000 UTC m=+527.177015063"
Sep 17 18:47:44.769305 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:44.769272    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-2rhkf"
Sep 17 18:47:51.753036 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:51.753008    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-4kkhn"
Sep 17 18:47:54.767361 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:47:54.767290    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-jtzqw"
Sep 17 18:48:48.712083 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:48.712054    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:48:48.714460 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:48.714434    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:48:48.715460 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:48.715441    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:48:48.718035 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:48.718015    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:48:54.458701 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.458666    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-chjq2"]
Sep 17 18:48:54.461878 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.461862    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:48:54.466295 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.466271    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z8znz\""
Sep 17 18:48:54.466434 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.466321    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:48:54.467203 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.467185    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:48:54.467319 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.467222    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:48:54.474701 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.474678    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-chjq2"]
Sep 17 18:48:54.605162 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.605126    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmkk\" (UniqueName: \"kubernetes.io/projected/6bccb4ef-e450-4e8d-b30f-f56a3754b53e-kube-api-access-npmkk\") pod \"seaweedfs-64568bcd49-chjq2\" (UID: \"6bccb4ef-e450-4e8d-b30f-f56a3754b53e\") " pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:48:54.605355 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.605190    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6bccb4ef-e450-4e8d-b30f-f56a3754b53e-data\") pod \"seaweedfs-64568bcd49-chjq2\" (UID: \"6bccb4ef-e450-4e8d-b30f-f56a3754b53e\") " pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:48:54.705679 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.705640    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmkk\" (UniqueName: \"kubernetes.io/projected/6bccb4ef-e450-4e8d-b30f-f56a3754b53e-kube-api-access-npmkk\") pod \"seaweedfs-64568bcd49-chjq2\" (UID: \"6bccb4ef-e450-4e8d-b30f-f56a3754b53e\") " pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:48:54.705856 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.705719    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6bccb4ef-e450-4e8d-b30f-f56a3754b53e-data\") pod \"seaweedfs-64568bcd49-chjq2\" (UID: \"6bccb4ef-e450-4e8d-b30f-f56a3754b53e\") " pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:48:54.706073 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.706056    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6bccb4ef-e450-4e8d-b30f-f56a3754b53e-data\") pod \"seaweedfs-64568bcd49-chjq2\" (UID: \"6bccb4ef-e450-4e8d-b30f-f56a3754b53e\") " pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:48:54.717490 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.717434    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmkk\" (UniqueName: \"kubernetes.io/projected/6bccb4ef-e450-4e8d-b30f-f56a3754b53e-kube-api-access-npmkk\") pod \"seaweedfs-64568bcd49-chjq2\" (UID: \"6bccb4ef-e450-4e8d-b30f-f56a3754b53e\") " pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:48:54.771113 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:54.771086    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:48:55.064000 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:55.063924    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-chjq2"]
Sep 17 18:48:55.067093 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:48:55.067069    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bccb4ef_e450_4e8d_b30f_f56a3754b53e.slice/crio-3a9400610860ef2df0c7a022bd634ac1364f27662f5664042e675532514738ae WatchSource:0}: Error finding container 3a9400610860ef2df0c7a022bd634ac1364f27662f5664042e675532514738ae: Status 404 returned error can't find the container with id 3a9400610860ef2df0c7a022bd634ac1364f27662f5664042e675532514738ae
Sep 17 18:48:56.070873 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:56.070839    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-chjq2" event={"ID":"6bccb4ef-e450-4e8d-b30f-f56a3754b53e","Type":"ContainerStarted","Data":"3a9400610860ef2df0c7a022bd634ac1364f27662f5664042e675532514738ae"}
Sep 17 18:48:58.080319 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:58.080275    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-chjq2" event={"ID":"6bccb4ef-e450-4e8d-b30f-f56a3754b53e","Type":"ContainerStarted","Data":"c01a956c44b78e2b3f2b775217efe18f61aca88804ea6a183066cdc417acafc3"}
Sep 17 18:48:58.080726 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:48:58.080349    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:49:04.086289 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:49:04.086201    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-chjq2"
Sep 17 18:49:04.105569 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:49:04.105522    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-chjq2" podStartSLOduration=7.377001061 podStartE2EDuration="10.105508048s" totalImagesPullingTime="2.728506987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:58.108262444 +0000 UTC m=+611.463521237" watchObservedRunningTime="2026-09-17 18:49:04.105508048 +0000 UTC m=+617.460766859"
Sep 17 18:50:12.256243 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.256197    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-j52ld"]
Sep 17 18:50:12.259546 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.259531    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:12.262686 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.262665    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 18:50:12.262791 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.262693    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-z89m6\""
Sep 17 18:50:12.270123 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.270102    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-j52ld"]
Sep 17 18:50:12.273911 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.273889    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-l64kh"]
Sep 17 18:50:12.276828 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.276813    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.279688 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.279670    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 18:50:12.279812 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.279796    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 18:50:12.281058 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.281043    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-s7wmz\""
Sep 17 18:50:12.288878 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.288853    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-l64kh"]
Sep 17 18:50:12.332956 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.332926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdhkl\" (UniqueName: \"kubernetes.io/projected/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-kube-api-access-hdhkl\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.333073 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.332967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-metrics-certs\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.333073 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.333004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65wx\" (UniqueName: \"kubernetes.io/projected/d88c8014-29b0-44f2-8fa0-854dce39b671-kube-api-access-c65wx\") pod \"model-serving-api-7b954ffc95-j52ld\" (UID: \"d88c8014-29b0-44f2-8fa0-854dce39b671\") " pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:12.333073 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.333021    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-cert\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.333073 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.333041    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88c8014-29b0-44f2-8fa0-854dce39b671-tls-certs\") pod \"model-serving-api-7b954ffc95-j52ld\" (UID: \"d88c8014-29b0-44f2-8fa0-854dce39b671\") " pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:12.434435 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.434402    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65wx\" (UniqueName: \"kubernetes.io/projected/d88c8014-29b0-44f2-8fa0-854dce39b671-kube-api-access-c65wx\") pod \"model-serving-api-7b954ffc95-j52ld\" (UID: \"d88c8014-29b0-44f2-8fa0-854dce39b671\") " pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:12.434435 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.434434    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-cert\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.434683 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.434456    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88c8014-29b0-44f2-8fa0-854dce39b671-tls-certs\") pod \"model-serving-api-7b954ffc95-j52ld\" (UID: \"d88c8014-29b0-44f2-8fa0-854dce39b671\") " pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:12.434683 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.434480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdhkl\" (UniqueName: \"kubernetes.io/projected/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-kube-api-access-hdhkl\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.434683 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.434516    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-metrics-certs\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.434683 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:50:12.434577    2557 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 18:50:12.434683 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:50:12.434639    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-cert podName:8a7204ed-f1a1-4d31-9c0c-74b9b14609ef nodeName:}" failed. No retries permitted until 2026-09-17 18:50:12.934622691 +0000 UTC m=+686.289881441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-cert") pod "odh-model-controller-7b966d7984-l64kh" (UID: "8a7204ed-f1a1-4d31-9c0c-74b9b14609ef") : secret "odh-model-controller-webhook-cert" not found
Sep 17 18:50:12.434683 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:50:12.434577    2557 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 18:50:12.434683 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:50:12.434671    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d88c8014-29b0-44f2-8fa0-854dce39b671-tls-certs podName:d88c8014-29b0-44f2-8fa0-854dce39b671 nodeName:}" failed. No retries permitted until 2026-09-17 18:50:12.934664564 +0000 UTC m=+686.289923315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d88c8014-29b0-44f2-8fa0-854dce39b671-tls-certs") pod "model-serving-api-7b954ffc95-j52ld" (UID: "d88c8014-29b0-44f2-8fa0-854dce39b671") : secret "model-serving-api-tls" not found
Sep 17 18:50:12.436997 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.436977    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-metrics-certs\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.452627 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.452603    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65wx\" (UniqueName: \"kubernetes.io/projected/d88c8014-29b0-44f2-8fa0-854dce39b671-kube-api-access-c65wx\") pod \"model-serving-api-7b954ffc95-j52ld\" (UID: \"d88c8014-29b0-44f2-8fa0-854dce39b671\") " pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:12.452737 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.452603    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdhkl\" (UniqueName: \"kubernetes.io/projected/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-kube-api-access-hdhkl\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.938797 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.938762    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-cert\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.938797 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.938801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88c8014-29b0-44f2-8fa0-854dce39b671-tls-certs\") pod \"model-serving-api-7b954ffc95-j52ld\" (UID: \"d88c8014-29b0-44f2-8fa0-854dce39b671\") " pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:12.941330 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.941297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7204ed-f1a1-4d31-9c0c-74b9b14609ef-cert\") pod \"odh-model-controller-7b966d7984-l64kh\" (UID: \"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef\") " pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:12.941456 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:12.941304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88c8014-29b0-44f2-8fa0-854dce39b671-tls-certs\") pod \"model-serving-api-7b954ffc95-j52ld\" (UID: \"d88c8014-29b0-44f2-8fa0-854dce39b671\") " pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:13.169961 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:13.169923    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:13.187063 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:13.187038    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:13.446058 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:13.446010    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-j52ld"]
Sep 17 18:50:13.447921 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:50:13.447893    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88c8014_29b0_44f2_8fa0_854dce39b671.slice/crio-9f5cafed691797cf6834c6bd3127bf2c8e60c0e6fce971ef46dc6147f0c06c23 WatchSource:0}: Error finding container 9f5cafed691797cf6834c6bd3127bf2c8e60c0e6fce971ef46dc6147f0c06c23: Status 404 returned error can't find the container with id 9f5cafed691797cf6834c6bd3127bf2c8e60c0e6fce971ef46dc6147f0c06c23
Sep 17 18:50:13.473722 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:13.473696    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-l64kh"]
Sep 17 18:50:13.476456 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:50:13.476423    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7204ed_f1a1_4d31_9c0c_74b9b14609ef.slice/crio-ed0bc5b95592691180ca96733d5929646c6e4bde391d3162a217d8220e1a80a7 WatchSource:0}: Error finding container ed0bc5b95592691180ca96733d5929646c6e4bde391d3162a217d8220e1a80a7: Status 404 returned error can't find the container with id ed0bc5b95592691180ca96733d5929646c6e4bde391d3162a217d8220e1a80a7
Sep 17 18:50:14.361548 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:14.361481    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-j52ld" event={"ID":"d88c8014-29b0-44f2-8fa0-854dce39b671","Type":"ContainerStarted","Data":"9f5cafed691797cf6834c6bd3127bf2c8e60c0e6fce971ef46dc6147f0c06c23"}
Sep 17 18:50:14.362896 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:14.362856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-l64kh" event={"ID":"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef","Type":"ContainerStarted","Data":"ed0bc5b95592691180ca96733d5929646c6e4bde391d3162a217d8220e1a80a7"}
Sep 17 18:50:17.378296 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:17.378252    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-j52ld" event={"ID":"d88c8014-29b0-44f2-8fa0-854dce39b671","Type":"ContainerStarted","Data":"7026d72be8c8af9e54f33b691702101f7fa9f112b76dfb8f866381bd64c6be72"}
Sep 17 18:50:17.378761 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:17.378360    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:50:17.379756 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:17.379735    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-l64kh" event={"ID":"8a7204ed-f1a1-4d31-9c0c-74b9b14609ef","Type":"ContainerStarted","Data":"6c5e4e35c1e1ec1babd6acff7f0943f92842f01e61b06cdefbceed5a1868cac0"}
Sep 17 18:50:17.379826 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:17.379782    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:17.437014 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:17.436961    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-j52ld" podStartSLOduration=1.640959383 podStartE2EDuration="5.436944583s" totalImagesPullingTime="3.7959852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:17.406621893 +0000 UTC m=+690.761880686" watchObservedRunningTime="2026-09-17 18:50:17.436944583 +0000 UTC m=+690.792203356"
Sep 17 18:50:17.437717 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:17.437686    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-l64kh" podStartSLOduration=1.625983478 podStartE2EDuration="5.437677563s" totalImagesPullingTime="3.811694085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:17.436772846 +0000 UTC m=+690.792031620" watchObservedRunningTime="2026-09-17 18:50:17.437677563 +0000 UTC m=+690.792936336"
Sep 17 18:50:28.386901 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:28.386868    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-l64kh"
Sep 17 18:50:28.388607 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:50:28.388589    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-j52ld"
Sep 17 18:52:01.265177 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.265099    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"]
Sep 17 18:52:01.268400 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.268378    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.271313 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.271287    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:52:01.271422 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.271286    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a006b-predictor-serving-cert\""
Sep 17 18:52:01.271422 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.271316    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:52:01.272430 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.272412    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b8646\""
Sep 17 18:52:01.272556 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.272435    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a006b-kube-rbac-proxy-sar-config\""
Sep 17 18:52:01.277420 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.277398    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"]
Sep 17 18:52:01.380285 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.380245    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/698c1003-1096-4588-a8af-9c8c93d260a3-proxy-tls\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.380488 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.380298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/698c1003-1096-4588-a8af-9c8c93d260a3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.380488 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.380368    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtqtc\" (UniqueName: \"kubernetes.io/projected/698c1003-1096-4588-a8af-9c8c93d260a3-kube-api-access-dtqtc\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.481870 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.481840    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/698c1003-1096-4588-a8af-9c8c93d260a3-proxy-tls\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.481870 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.481875    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/698c1003-1096-4588-a8af-9c8c93d260a3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.482099 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.481892    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqtc\" (UniqueName: \"kubernetes.io/projected/698c1003-1096-4588-a8af-9c8c93d260a3-kube-api-access-dtqtc\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.482099 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:52:01.482014    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a006b-predictor-serving-cert: secret "success-200-isvc-a006b-predictor-serving-cert" not found
Sep 17 18:52:01.482099 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:52:01.482098    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/698c1003-1096-4588-a8af-9c8c93d260a3-proxy-tls podName:698c1003-1096-4588-a8af-9c8c93d260a3 nodeName:}" failed. No retries permitted until 2026-09-17 18:52:01.982079166 +0000 UTC m=+795.337337924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/698c1003-1096-4588-a8af-9c8c93d260a3-proxy-tls") pod "success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" (UID: "698c1003-1096-4588-a8af-9c8c93d260a3") : secret "success-200-isvc-a006b-predictor-serving-cert" not found
Sep 17 18:52:01.482550 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.482529    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/698c1003-1096-4588-a8af-9c8c93d260a3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.491776 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.491752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqtc\" (UniqueName: \"kubernetes.io/projected/698c1003-1096-4588-a8af-9c8c93d260a3-kube-api-access-dtqtc\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.986882 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.986848    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/698c1003-1096-4588-a8af-9c8c93d260a3-proxy-tls\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:01.989269 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:01.989240    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/698c1003-1096-4588-a8af-9c8c93d260a3-proxy-tls\") pod \"success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") " pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:02.179346 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:02.179313    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:02.454095 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:02.454063    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"]
Sep 17 18:52:02.456524 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:52:02.456498    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698c1003_1096_4588_a8af_9c8c93d260a3.slice/crio-60453343b450c6f3c83ad1b4ebb87e329072e053cc364be8f25377db0612ebb0 WatchSource:0}: Error finding container 60453343b450c6f3c83ad1b4ebb87e329072e053cc364be8f25377db0612ebb0: Status 404 returned error can't find the container with id 60453343b450c6f3c83ad1b4ebb87e329072e053cc364be8f25377db0612ebb0
Sep 17 18:52:02.458552 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:02.458535    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:52:02.780971 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:02.780884    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" event={"ID":"698c1003-1096-4588-a8af-9c8c93d260a3","Type":"ContainerStarted","Data":"60453343b450c6f3c83ad1b4ebb87e329072e053cc364be8f25377db0612ebb0"}
Sep 17 18:52:14.836792 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:14.836760    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" event={"ID":"698c1003-1096-4588-a8af-9c8c93d260a3","Type":"ContainerStarted","Data":"af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025"}
Sep 17 18:52:17.851280 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:17.851242    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" event={"ID":"698c1003-1096-4588-a8af-9c8c93d260a3","Type":"ContainerStarted","Data":"e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1"}
Sep 17 18:52:17.851738 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:17.851327    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:17.871010 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:17.870961    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" podStartSLOduration=2.281486819 podStartE2EDuration="16.87094732s" totalImagesPullingTime="14.589460501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:52:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:52:17.869385732 +0000 UTC m=+811.224644533" watchObservedRunningTime="2026-09-17 18:52:17.87094732 +0000 UTC m=+811.226206093"
Sep 17 18:52:18.855503 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:18.855474    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:18.856578 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:18.856550    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:52:19.858883 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:19.858841    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:52:24.864039 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:24.864003    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:52:24.864579 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:24.864547    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:52:34.865508 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:34.865468    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:52:44.864835 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:44.864795    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:52:54.865242 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:52:54.865186    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 18:53:04.865317 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:04.865285    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:53:21.057520 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.057484    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"]
Sep 17 18:53:21.060602 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.060587    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:21.063583 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.063563    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a006b-kube-rbac-proxy-sar-config\""
Sep 17 18:53:21.064179 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.064165    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-a006b-serving-cert\""
Sep 17 18:53:21.078318 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.078297    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"]
Sep 17 18:53:21.102146 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.102123    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957f182e-1690-4a58-92c6-ca929e2fea7b-proxy-tls\") pod \"switch-graph-a006b-757f96b786-hwwfz\" (UID: \"957f182e-1690-4a58-92c6-ca929e2fea7b\") " pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:21.102268 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.102164    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f182e-1690-4a58-92c6-ca929e2fea7b-openshift-service-ca-bundle\") pod \"switch-graph-a006b-757f96b786-hwwfz\" (UID: \"957f182e-1690-4a58-92c6-ca929e2fea7b\") " pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:21.202740 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.202700    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957f182e-1690-4a58-92c6-ca929e2fea7b-proxy-tls\") pod \"switch-graph-a006b-757f96b786-hwwfz\" (UID: \"957f182e-1690-4a58-92c6-ca929e2fea7b\") " pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:21.202905 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.202752    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f182e-1690-4a58-92c6-ca929e2fea7b-openshift-service-ca-bundle\") pod \"switch-graph-a006b-757f96b786-hwwfz\" (UID: \"957f182e-1690-4a58-92c6-ca929e2fea7b\") " pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:21.203521 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.203498    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f182e-1690-4a58-92c6-ca929e2fea7b-openshift-service-ca-bundle\") pod \"switch-graph-a006b-757f96b786-hwwfz\" (UID: \"957f182e-1690-4a58-92c6-ca929e2fea7b\") " pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:21.205119 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.205102    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957f182e-1690-4a58-92c6-ca929e2fea7b-proxy-tls\") pod \"switch-graph-a006b-757f96b786-hwwfz\" (UID: \"957f182e-1690-4a58-92c6-ca929e2fea7b\") " pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:21.369900 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.369872    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:21.639761 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:21.639735    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"]
Sep 17 18:53:21.641853 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:53:21.641816    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957f182e_1690_4a58_92c6_ca929e2fea7b.slice/crio-e929fba61cd20111e64b17574ef009f0ffbfd9fce349f5f046b595da555375b0 WatchSource:0}: Error finding container e929fba61cd20111e64b17574ef009f0ffbfd9fce349f5f046b595da555375b0: Status 404 returned error can't find the container with id e929fba61cd20111e64b17574ef009f0ffbfd9fce349f5f046b595da555375b0
Sep 17 18:53:22.092929 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:22.092897    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" event={"ID":"957f182e-1690-4a58-92c6-ca929e2fea7b","Type":"ContainerStarted","Data":"e929fba61cd20111e64b17574ef009f0ffbfd9fce349f5f046b595da555375b0"}
Sep 17 18:53:25.109407 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:25.109369    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" event={"ID":"957f182e-1690-4a58-92c6-ca929e2fea7b","Type":"ContainerStarted","Data":"cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0"}
Sep 17 18:53:25.109846 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:25.109496    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:25.128494 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:25.128437    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" podStartSLOduration=1.716456915 podStartE2EDuration="4.128422662s" totalImagesPullingTime="2.411965747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:25.127298346 +0000 UTC m=+878.482557139" watchObservedRunningTime="2026-09-17 18:53:25.128422662 +0000 UTC m=+878.483681472"
Sep 17 18:53:31.118934 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:31.118858    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:35.289515 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.289483    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"]
Sep 17 18:53:35.289987 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.289667    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b" containerID="cri-o://cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0" gracePeriod=30
Sep 17 18:53:35.438180 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.438150    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"]
Sep 17 18:53:35.438408 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.438380    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container" containerID="cri-o://af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025" gracePeriod=30
Sep 17 18:53:35.438536 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.438423    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kube-rbac-proxy" containerID="cri-o://e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1" gracePeriod=30
Sep 17 18:53:35.597118 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.597083    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"]
Sep 17 18:53:35.600431 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.600414    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.603284 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.603261    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7989f-kube-rbac-proxy-sar-config\""
Sep 17 18:53:35.603412 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.603397    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7989f-predictor-serving-cert\""
Sep 17 18:53:35.616671 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.616643    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"]
Sep 17 18:53:35.719677 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.719643    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af3f40cd-af17-4bb0-9127-09346f9e1954-proxy-tls\") pod \"success-200-isvc-7989f-predictor-749fdfd4db-6qthz\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") " pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.719854 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.719744    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7989f-predictor-749fdfd4db-6qthz\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") " pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.719854 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.719796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfx5z\" (UniqueName: \"kubernetes.io/projected/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-api-access-kfx5z\") pod \"success-200-isvc-7989f-predictor-749fdfd4db-6qthz\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") " pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.820994 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.820960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7989f-predictor-749fdfd4db-6qthz\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") " pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.821179 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.821006    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfx5z\" (UniqueName: \"kubernetes.io/projected/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-api-access-kfx5z\") pod \"success-200-isvc-7989f-predictor-749fdfd4db-6qthz\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") " pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.821179 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.821053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af3f40cd-af17-4bb0-9127-09346f9e1954-proxy-tls\") pod \"success-200-isvc-7989f-predictor-749fdfd4db-6qthz\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") " pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.821679 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.821659    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7989f-predictor-749fdfd4db-6qthz\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") " pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.823534 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.823510    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af3f40cd-af17-4bb0-9127-09346f9e1954-proxy-tls\") pod \"success-200-isvc-7989f-predictor-749fdfd4db-6qthz\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") " pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.829024 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.829000    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfx5z\" (UniqueName: \"kubernetes.io/projected/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-api-access-kfx5z\") pod \"success-200-isvc-7989f-predictor-749fdfd4db-6qthz\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") " pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:35.910361 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:35.910326    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:36.116991 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:36.116904    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:53:36.159539 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:36.159492    2557 generic.go:356] "Generic (PLEG): container finished" podID="698c1003-1096-4588-a8af-9c8c93d260a3" containerID="e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1" exitCode=2
Sep 17 18:53:36.159726 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:36.159564    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" event={"ID":"698c1003-1096-4588-a8af-9c8c93d260a3","Type":"ContainerDied","Data":"e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1"}
Sep 17 18:53:36.182538 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:36.182505    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"]
Sep 17 18:53:36.185947 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:53:36.185925    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3f40cd_af17_4bb0_9127_09346f9e1954.slice/crio-39d11fc975fcd6c1cd271e72cbd3194bec88883b3731ae6d909adf5d724e14cd WatchSource:0}: Error finding container 39d11fc975fcd6c1cd271e72cbd3194bec88883b3731ae6d909adf5d724e14cd: Status 404 returned error can't find the container with id 39d11fc975fcd6c1cd271e72cbd3194bec88883b3731ae6d909adf5d724e14cd
Sep 17 18:53:37.165395 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:37.165361    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" event={"ID":"af3f40cd-af17-4bb0-9127-09346f9e1954","Type":"ContainerStarted","Data":"5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a"}
Sep 17 18:53:37.165839 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:37.165417    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" event={"ID":"af3f40cd-af17-4bb0-9127-09346f9e1954","Type":"ContainerStarted","Data":"83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632"}
Sep 17 18:53:37.165839 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:37.165441    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" event={"ID":"af3f40cd-af17-4bb0-9127-09346f9e1954","Type":"ContainerStarted","Data":"39d11fc975fcd6c1cd271e72cbd3194bec88883b3731ae6d909adf5d724e14cd"}
Sep 17 18:53:37.165839 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:37.165529    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:37.165839 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:37.165580    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:37.167225 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:37.167184    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:53:37.189330 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:37.189292    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podStartSLOduration=2.189276981 podStartE2EDuration="2.189276981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:37.188513933 +0000 UTC m=+890.543772724" watchObservedRunningTime="2026-09-17 18:53:37.189276981 +0000 UTC m=+890.544535753"
Sep 17 18:53:38.169086 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.169051    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:53:38.584106 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.584083    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:53:38.642257 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.642226    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/698c1003-1096-4588-a8af-9c8c93d260a3-kube-rbac-proxy-sar-config\") pod \"698c1003-1096-4588-a8af-9c8c93d260a3\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") "
Sep 17 18:53:38.642436 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.642304    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/698c1003-1096-4588-a8af-9c8c93d260a3-proxy-tls\") pod \"698c1003-1096-4588-a8af-9c8c93d260a3\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") "
Sep 17 18:53:38.642436 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.642330    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtqtc\" (UniqueName: \"kubernetes.io/projected/698c1003-1096-4588-a8af-9c8c93d260a3-kube-api-access-dtqtc\") pod \"698c1003-1096-4588-a8af-9c8c93d260a3\" (UID: \"698c1003-1096-4588-a8af-9c8c93d260a3\") "
Sep 17 18:53:38.642590 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.642566    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698c1003-1096-4588-a8af-9c8c93d260a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "698c1003-1096-4588-a8af-9c8c93d260a3" (UID: "698c1003-1096-4588-a8af-9c8c93d260a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:38.644416 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.644385    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698c1003-1096-4588-a8af-9c8c93d260a3-kube-api-access-dtqtc" (OuterVolumeSpecName: "kube-api-access-dtqtc") pod "698c1003-1096-4588-a8af-9c8c93d260a3" (UID: "698c1003-1096-4588-a8af-9c8c93d260a3"). InnerVolumeSpecName "kube-api-access-dtqtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:38.644531 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.644415    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698c1003-1096-4588-a8af-9c8c93d260a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "698c1003-1096-4588-a8af-9c8c93d260a3" (UID: "698c1003-1096-4588-a8af-9c8c93d260a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:38.742829 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.742743    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtqtc\" (UniqueName: \"kubernetes.io/projected/698c1003-1096-4588-a8af-9c8c93d260a3-kube-api-access-dtqtc\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:53:38.742829 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.742773    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/698c1003-1096-4588-a8af-9c8c93d260a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:53:38.742829 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:38.742784    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/698c1003-1096-4588-a8af-9c8c93d260a3-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:53:39.174588 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.174551    2557 generic.go:356] "Generic (PLEG): container finished" podID="698c1003-1096-4588-a8af-9c8c93d260a3" containerID="af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025" exitCode=0
Sep 17 18:53:39.175072 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.174603    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"
Sep 17 18:53:39.175072 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.174610    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" event={"ID":"698c1003-1096-4588-a8af-9c8c93d260a3","Type":"ContainerDied","Data":"af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025"}
Sep 17 18:53:39.175072 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.174659    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl" event={"ID":"698c1003-1096-4588-a8af-9c8c93d260a3","Type":"ContainerDied","Data":"60453343b450c6f3c83ad1b4ebb87e329072e053cc364be8f25377db0612ebb0"}
Sep 17 18:53:39.175072 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.174685    2557 scope.go:117] "RemoveContainer" containerID="e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1"
Sep 17 18:53:39.182526 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.182509    2557 scope.go:117] "RemoveContainer" containerID="af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025"
Sep 17 18:53:39.189571 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.189554    2557 scope.go:117] "RemoveContainer" containerID="e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1"
Sep 17 18:53:39.189814 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:53:39.189797    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1\": container with ID starting with e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1 not found: ID does not exist" containerID="e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1"
Sep 17 18:53:39.189870 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.189822    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1"} err="failed to get container status \"e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1\": rpc error: code = NotFound desc = could not find container \"e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1\": container with ID starting with e424a28b105564c11718c4b7d1071cb375fd1d74c38fecfc6664b02b79d1add1 not found: ID does not exist"
Sep 17 18:53:39.189870 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.189839    2557 scope.go:117] "RemoveContainer" containerID="af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025"
Sep 17 18:53:39.190088 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:53:39.190072    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025\": container with ID starting with af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025 not found: ID does not exist" containerID="af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025"
Sep 17 18:53:39.190137 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.190090    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025"} err="failed to get container status \"af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025\": rpc error: code = NotFound desc = could not find container \"af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025\": container with ID starting with af1c52edb5a60bc2d65c941ce063530a1e0d1dedf319bfebe7484eae95981025 not found: ID does not exist"
Sep 17 18:53:39.194078 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.194047    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"]
Sep 17 18:53:39.198102 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:39.198075    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a006b-predictor-6cfbdfb69d-g6ltl"]
Sep 17 18:53:40.835693 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:40.835661    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" path="/var/lib/kubelet/pods/698c1003-1096-4588-a8af-9c8c93d260a3/volumes"
Sep 17 18:53:41.117118 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:41.117032    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:53:43.175291 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:43.175262    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:53:43.175882 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:43.175857    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:53:46.117848 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:46.117811    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:53:46.118346 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:46.117942    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:53:48.743470 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:48.743438    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:53:48.743951 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:48.743930    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:53:48.746710 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:48.746689    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:53:48.747083 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:48.747064    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:53:51.117727 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:51.117688    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:53:53.175908 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:53.175871    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:53:56.116939 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:53:56.116902    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:01.118155 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:01.118093    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:03.176420 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:03.176383    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:54:05.434536 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:05.434512    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:54:05.549148 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:05.549055    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f182e-1690-4a58-92c6-ca929e2fea7b-openshift-service-ca-bundle\") pod \"957f182e-1690-4a58-92c6-ca929e2fea7b\" (UID: \"957f182e-1690-4a58-92c6-ca929e2fea7b\") "
Sep 17 18:54:05.549148 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:05.549090    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957f182e-1690-4a58-92c6-ca929e2fea7b-proxy-tls\") pod \"957f182e-1690-4a58-92c6-ca929e2fea7b\" (UID: \"957f182e-1690-4a58-92c6-ca929e2fea7b\") "
Sep 17 18:54:05.549514 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:05.549488    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957f182e-1690-4a58-92c6-ca929e2fea7b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "957f182e-1690-4a58-92c6-ca929e2fea7b" (UID: "957f182e-1690-4a58-92c6-ca929e2fea7b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:05.551242 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:05.551207    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957f182e-1690-4a58-92c6-ca929e2fea7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "957f182e-1690-4a58-92c6-ca929e2fea7b" (UID: "957f182e-1690-4a58-92c6-ca929e2fea7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:05.649882 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:05.649850    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f182e-1690-4a58-92c6-ca929e2fea7b-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:54:05.649882 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:05.649878    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/957f182e-1690-4a58-92c6-ca929e2fea7b-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.276255 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.276195    2557 generic.go:356] "Generic (PLEG): container finished" podID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerID="cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0" exitCode=0
Sep 17 18:54:06.276255 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.276256    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"
Sep 17 18:54:06.276515 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.276278    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" event={"ID":"957f182e-1690-4a58-92c6-ca929e2fea7b","Type":"ContainerDied","Data":"cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0"}
Sep 17 18:54:06.276515 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.276332    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz" event={"ID":"957f182e-1690-4a58-92c6-ca929e2fea7b","Type":"ContainerDied","Data":"e929fba61cd20111e64b17574ef009f0ffbfd9fce349f5f046b595da555375b0"}
Sep 17 18:54:06.276515 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.276355    2557 scope.go:117] "RemoveContainer" containerID="cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0"
Sep 17 18:54:06.284587 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.284571    2557 scope.go:117] "RemoveContainer" containerID="cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0"
Sep 17 18:54:06.284836 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:54:06.284818    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0\": container with ID starting with cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0 not found: ID does not exist" containerID="cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0"
Sep 17 18:54:06.284908 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.284842    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0"} err="failed to get container status \"cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0\": rpc error: code = NotFound desc = could not find container \"cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0\": container with ID starting with cd476e967ef3fea09a72299c80a0f2576fc1e16ae168a1e9748ece27f850c4d0 not found: ID does not exist"
Sep 17 18:54:06.297724 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.297701    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"]
Sep 17 18:54:06.301383 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.301362    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-a006b-757f96b786-hwwfz"]
Sep 17 18:54:06.835311 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:06.835279    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" path="/var/lib/kubelet/pods/957f182e-1690-4a58-92c6-ca929e2fea7b/volumes"
Sep 17 18:54:11.035891 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.035857    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"]
Sep 17 18:54:11.036282 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.036171    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kube-rbac-proxy"
Sep 17 18:54:11.036282 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.036182    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kube-rbac-proxy"
Sep 17 18:54:11.036282 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.036192    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b"
Sep 17 18:54:11.036282 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.036197    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b"
Sep 17 18:54:11.036282 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.036222    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container"
Sep 17 18:54:11.036282 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.036228    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container"
Sep 17 18:54:11.036282 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.036273    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="957f182e-1690-4a58-92c6-ca929e2fea7b" containerName="switch-graph-a006b"
Sep 17 18:54:11.036282 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.036283    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kserve-container"
Sep 17 18:54:11.036578 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.036291    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="698c1003-1096-4588-a8af-9c8c93d260a3" containerName="kube-rbac-proxy"
Sep 17 18:54:11.040530 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.040509    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:11.043155 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.043137    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Sep 17 18:54:11.043362 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.043349    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 17 18:54:11.048772 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.048748    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"]
Sep 17 18:54:11.193741 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.193702    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-openshift-service-ca-bundle\") pod \"model-chainer-678bbd4574-qz25r\" (UID: \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\") " pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:11.193925 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.193759    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-proxy-tls\") pod \"model-chainer-678bbd4574-qz25r\" (UID: \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\") " pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:11.294253 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.294151    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-openshift-service-ca-bundle\") pod \"model-chainer-678bbd4574-qz25r\" (UID: \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\") " pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:11.294253 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.294204    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-proxy-tls\") pod \"model-chainer-678bbd4574-qz25r\" (UID: \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\") " pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:11.294436 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:54:11.294341    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Sep 17 18:54:11.294436 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:54:11.294396    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-proxy-tls podName:3bf113d4-a332-46b7-aa6d-6c272f6fd8ea nodeName:}" failed. No retries permitted until 2026-09-17 18:54:11.794380161 +0000 UTC m=+925.149638912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-proxy-tls") pod "model-chainer-678bbd4574-qz25r" (UID: "3bf113d4-a332-46b7-aa6d-6c272f6fd8ea") : secret "model-chainer-serving-cert" not found
Sep 17 18:54:11.294944 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.294920    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-openshift-service-ca-bundle\") pod \"model-chainer-678bbd4574-qz25r\" (UID: \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\") " pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:11.798313 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.798274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-proxy-tls\") pod \"model-chainer-678bbd4574-qz25r\" (UID: \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\") " pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:11.800668 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.800638    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-proxy-tls\") pod \"model-chainer-678bbd4574-qz25r\" (UID: \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\") " pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:11.951868 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:11.951831    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:12.224986 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:12.224961    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"]
Sep 17 18:54:12.227127 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:54:12.227102    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf113d4_a332_46b7_aa6d_6c272f6fd8ea.slice/crio-0a51fd26ae207fcc453cbeab19988066adea759c6551bb8ad82c5e40a9cd79e6 WatchSource:0}: Error finding container 0a51fd26ae207fcc453cbeab19988066adea759c6551bb8ad82c5e40a9cd79e6: Status 404 returned error can't find the container with id 0a51fd26ae207fcc453cbeab19988066adea759c6551bb8ad82c5e40a9cd79e6
Sep 17 18:54:12.303668 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:12.303635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" event={"ID":"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea","Type":"ContainerStarted","Data":"0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377"}
Sep 17 18:54:12.303832 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:12.303685    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" event={"ID":"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea","Type":"ContainerStarted","Data":"0a51fd26ae207fcc453cbeab19988066adea759c6551bb8ad82c5e40a9cd79e6"}
Sep 17 18:54:12.303832 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:12.303754    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:12.327684 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:12.327639    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" podStartSLOduration=1.327624328 podStartE2EDuration="1.327624328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:12.325434586 +0000 UTC m=+925.680693371" watchObservedRunningTime="2026-09-17 18:54:12.327624328 +0000 UTC m=+925.682883100"
Sep 17 18:54:13.176392 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:13.176359    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 18:54:18.313606 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:18.313569    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:21.610895 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:21.610855    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"]
Sep 17 18:54:21.611343 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:21.611121    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer" containerID="cri-o://0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377" gracePeriod=30
Sep 17 18:54:21.924965 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:21.924888    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"]
Sep 17 18:54:21.928356 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:21.928336    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:21.931318 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:21.931293    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-887e9-kube-rbac-proxy-sar-config\""
Sep 17 18:54:21.931431 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:21.931412    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-887e9-predictor-serving-cert\""
Sep 17 18:54:21.939972 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:21.939948    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"]
Sep 17 18:54:22.082436 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.082403    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-887e9-predictor-7989ccbd74-bm7jl\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") " pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.082629 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.082454    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wg7w\" (UniqueName: \"kubernetes.io/projected/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-api-access-5wg7w\") pod \"success-200-isvc-887e9-predictor-7989ccbd74-bm7jl\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") " pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.082629 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.082532    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-proxy-tls\") pod \"success-200-isvc-887e9-predictor-7989ccbd74-bm7jl\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") " pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.183711 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.183626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-887e9-predictor-7989ccbd74-bm7jl\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") " pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.183711 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.183676    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wg7w\" (UniqueName: \"kubernetes.io/projected/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-api-access-5wg7w\") pod \"success-200-isvc-887e9-predictor-7989ccbd74-bm7jl\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") " pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.183959 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.183711    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-proxy-tls\") pod \"success-200-isvc-887e9-predictor-7989ccbd74-bm7jl\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") " pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.184312 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.184288    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-887e9-predictor-7989ccbd74-bm7jl\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") " pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.186137 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.186115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-proxy-tls\") pod \"success-200-isvc-887e9-predictor-7989ccbd74-bm7jl\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") " pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.196884 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.196863    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wg7w\" (UniqueName: \"kubernetes.io/projected/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-api-access-5wg7w\") pod \"success-200-isvc-887e9-predictor-7989ccbd74-bm7jl\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") " pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.238417 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.238392    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:22.511494 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:22.511468    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"]
Sep 17 18:54:22.513681 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:54:22.513654    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29491bc1_33cb_41d4_8d5a_6b94f5d4f8d0.slice/crio-14a0a63dfd2525c09671bf3c2c97d0e3e274de2afe4e84832f56e86e3ba1ca9c WatchSource:0}: Error finding container 14a0a63dfd2525c09671bf3c2c97d0e3e274de2afe4e84832f56e86e3ba1ca9c: Status 404 returned error can't find the container with id 14a0a63dfd2525c09671bf3c2c97d0e3e274de2afe4e84832f56e86e3ba1ca9c
Sep 17 18:54:23.176940 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:23.176908    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 18:54:23.311887 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:23.311847    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:23.350290 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:23.350250    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" event={"ID":"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0","Type":"ContainerStarted","Data":"9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff"}
Sep 17 18:54:23.350480 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:23.350350    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:23.350480 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:23.350380    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" event={"ID":"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0","Type":"ContainerStarted","Data":"851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219"}
Sep 17 18:54:23.350480 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:23.350406    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:23.350480 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:23.350424    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" event={"ID":"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0","Type":"ContainerStarted","Data":"14a0a63dfd2525c09671bf3c2c97d0e3e274de2afe4e84832f56e86e3ba1ca9c"}
Sep 17 18:54:23.351898 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:23.351870    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:54:23.371000 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:23.370949    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podStartSLOduration=2.370935885 podStartE2EDuration="2.370935885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:23.37034077 +0000 UTC m=+936.725599565" watchObservedRunningTime="2026-09-17 18:54:23.370935885 +0000 UTC m=+936.726194716"
Sep 17 18:54:24.354296 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:24.354253    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:54:28.312494 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:28.312457    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:29.359444 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:29.359417    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:54:29.359855 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:29.359832    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:54:33.312079 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:33.312039    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:33.312517 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:33.312134    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:35.538076 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.538042    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"]
Sep 17 18:54:35.541262 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.541246    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:35.543844 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.543822    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7989f-serving-cert\""
Sep 17 18:54:35.543963 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.543907    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7989f-kube-rbac-proxy-sar-config\""
Sep 17 18:54:35.550155 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.550132    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"]
Sep 17 18:54:35.587569 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.587537    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3eac444-40fd-4d8e-b383-c1db59ddc96f-openshift-service-ca-bundle\") pod \"switch-graph-7989f-64d5897b47-4j958\" (UID: \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\") " pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:35.587712 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.587582    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3eac444-40fd-4d8e-b383-c1db59ddc96f-proxy-tls\") pod \"switch-graph-7989f-64d5897b47-4j958\" (UID: \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\") " pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:35.688016 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.687975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3eac444-40fd-4d8e-b383-c1db59ddc96f-proxy-tls\") pod \"switch-graph-7989f-64d5897b47-4j958\" (UID: \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\") " pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:35.688171 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.688091    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3eac444-40fd-4d8e-b383-c1db59ddc96f-openshift-service-ca-bundle\") pod \"switch-graph-7989f-64d5897b47-4j958\" (UID: \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\") " pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:35.688171 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:54:35.688119    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-7989f-serving-cert: secret "switch-graph-7989f-serving-cert" not found
Sep 17 18:54:35.688344 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:54:35.688182    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3eac444-40fd-4d8e-b383-c1db59ddc96f-proxy-tls podName:f3eac444-40fd-4d8e-b383-c1db59ddc96f nodeName:}" failed. No retries permitted until 2026-09-17 18:54:36.188164801 +0000 UTC m=+949.543423551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f3eac444-40fd-4d8e-b383-c1db59ddc96f-proxy-tls") pod "switch-graph-7989f-64d5897b47-4j958" (UID: "f3eac444-40fd-4d8e-b383-c1db59ddc96f") : secret "switch-graph-7989f-serving-cert" not found
Sep 17 18:54:35.688795 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:35.688773    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3eac444-40fd-4d8e-b383-c1db59ddc96f-openshift-service-ca-bundle\") pod \"switch-graph-7989f-64d5897b47-4j958\" (UID: \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\") " pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:36.191356 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:36.191322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3eac444-40fd-4d8e-b383-c1db59ddc96f-proxy-tls\") pod \"switch-graph-7989f-64d5897b47-4j958\" (UID: \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\") " pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:36.193805 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:36.193773    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3eac444-40fd-4d8e-b383-c1db59ddc96f-proxy-tls\") pod \"switch-graph-7989f-64d5897b47-4j958\" (UID: \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\") " pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:36.452047 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:36.451965    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:36.726531 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:36.726504    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"]
Sep 17 18:54:36.728181 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:54:36.728152    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3eac444_40fd_4d8e_b383_c1db59ddc96f.slice/crio-d9d617e6353ab5d7f2f02401256b2a8021823f00f009aa14db65b6e5a8004014 WatchSource:0}: Error finding container d9d617e6353ab5d7f2f02401256b2a8021823f00f009aa14db65b6e5a8004014: Status 404 returned error can't find the container with id d9d617e6353ab5d7f2f02401256b2a8021823f00f009aa14db65b6e5a8004014
Sep 17 18:54:37.405687 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:37.405652    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" event={"ID":"f3eac444-40fd-4d8e-b383-c1db59ddc96f","Type":"ContainerStarted","Data":"ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa"}
Sep 17 18:54:37.405915 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:37.405699    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" event={"ID":"f3eac444-40fd-4d8e-b383-c1db59ddc96f","Type":"ContainerStarted","Data":"d9d617e6353ab5d7f2f02401256b2a8021823f00f009aa14db65b6e5a8004014"}
Sep 17 18:54:37.405915 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:37.405731    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:37.425763 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:37.425715    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" podStartSLOduration=2.425700475 podStartE2EDuration="2.425700475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:37.425149823 +0000 UTC m=+950.780408628" watchObservedRunningTime="2026-09-17 18:54:37.425700475 +0000 UTC m=+950.780959301"
Sep 17 18:54:38.312301 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:38.312262    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:39.360339 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:39.360297    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:54:43.311832 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:43.311794    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:43.415775 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:43.415745    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 18:54:48.312496 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:48.312441    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:54:49.360464 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:49.360420    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:54:51.758562 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:51.758539    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:51.796194 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:51.796164    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-openshift-service-ca-bundle\") pod \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\" (UID: \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\") "
Sep 17 18:54:51.796344 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:51.796231    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-proxy-tls\") pod \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\" (UID: \"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea\") "
Sep 17 18:54:51.796613 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:51.796582    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" (UID: "3bf113d4-a332-46b7-aa6d-6c272f6fd8ea"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:51.798284 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:51.798259    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" (UID: "3bf113d4-a332-46b7-aa6d-6c272f6fd8ea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:51.896745 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:51.896648    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:54:51.896745 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:51.896689    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 18:54:52.465344 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.465307    2557 generic.go:356] "Generic (PLEG): container finished" podID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerID="0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377" exitCode=0
Sep 17 18:54:52.465553 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.465344    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" event={"ID":"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea","Type":"ContainerDied","Data":"0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377"}
Sep 17 18:54:52.465553 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.465364    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"
Sep 17 18:54:52.465553 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.465415    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r" event={"ID":"3bf113d4-a332-46b7-aa6d-6c272f6fd8ea","Type":"ContainerDied","Data":"0a51fd26ae207fcc453cbeab19988066adea759c6551bb8ad82c5e40a9cd79e6"}
Sep 17 18:54:52.465553 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.465446    2557 scope.go:117] "RemoveContainer" containerID="0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377"
Sep 17 18:54:52.473805 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.473787    2557 scope.go:117] "RemoveContainer" containerID="0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377"
Sep 17 18:54:52.474079 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:54:52.474061    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377\": container with ID starting with 0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377 not found: ID does not exist" containerID="0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377"
Sep 17 18:54:52.474128 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.474086    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377"} err="failed to get container status \"0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377\": rpc error: code = NotFound desc = could not find container \"0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377\": container with ID starting with 0851375915eca3395d37734ea9c3593fc8e6ec9fe0ea3abc95ac2324596bf377 not found: ID does not exist"
Sep 17 18:54:52.488175 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.488152    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"]
Sep 17 18:54:52.497383 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.497364    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-678bbd4574-qz25r"]
Sep 17 18:54:52.836345 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:52.836318    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" path="/var/lib/kubelet/pods/3bf113d4-a332-46b7-aa6d-6c272f6fd8ea/volumes"
Sep 17 18:54:59.360581 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:54:59.360544    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:55:09.361081 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:09.361054    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 18:55:21.869827 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.869793    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"]
Sep 17 18:55:21.870310 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.870132    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer"
Sep 17 18:55:21.870310 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.870145    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer"
Sep 17 18:55:21.870310 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.870222    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf113d4-a332-46b7-aa6d-6c272f6fd8ea" containerName="model-chainer"
Sep 17 18:55:21.873047 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.873033    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:21.875813 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.875794    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-887e9-kube-rbac-proxy-sar-config\""
Sep 17 18:55:21.875886 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.875810    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-887e9-serving-cert\""
Sep 17 18:55:21.886613 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.886589    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"]
Sep 17 18:55:21.919034 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.919004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32d95902-6260-46cb-abb8-d103e438b80b-proxy-tls\") pod \"sequence-graph-887e9-57cbf8598b-fvfbt\" (UID: \"32d95902-6260-46cb-abb8-d103e438b80b\") " pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:21.919206 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:21.919067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d95902-6260-46cb-abb8-d103e438b80b-openshift-service-ca-bundle\") pod \"sequence-graph-887e9-57cbf8598b-fvfbt\" (UID: \"32d95902-6260-46cb-abb8-d103e438b80b\") " pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:22.019550 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:22.019514    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32d95902-6260-46cb-abb8-d103e438b80b-proxy-tls\") pod \"sequence-graph-887e9-57cbf8598b-fvfbt\" (UID: \"32d95902-6260-46cb-abb8-d103e438b80b\") " pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:22.019749 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:22.019598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d95902-6260-46cb-abb8-d103e438b80b-openshift-service-ca-bundle\") pod \"sequence-graph-887e9-57cbf8598b-fvfbt\" (UID: \"32d95902-6260-46cb-abb8-d103e438b80b\") " pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:22.019749 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:55:22.019658    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-887e9-serving-cert: secret "sequence-graph-887e9-serving-cert" not found
Sep 17 18:55:22.019749 ip-10-0-139-5 kubenswrapper[2557]: E0917 18:55:22.019726    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d95902-6260-46cb-abb8-d103e438b80b-proxy-tls podName:32d95902-6260-46cb-abb8-d103e438b80b nodeName:}" failed. No retries permitted until 2026-09-17 18:55:22.519709511 +0000 UTC m=+995.874968279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/32d95902-6260-46cb-abb8-d103e438b80b-proxy-tls") pod "sequence-graph-887e9-57cbf8598b-fvfbt" (UID: "32d95902-6260-46cb-abb8-d103e438b80b") : secret "sequence-graph-887e9-serving-cert" not found
Sep 17 18:55:22.020305 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:22.020283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d95902-6260-46cb-abb8-d103e438b80b-openshift-service-ca-bundle\") pod \"sequence-graph-887e9-57cbf8598b-fvfbt\" (UID: \"32d95902-6260-46cb-abb8-d103e438b80b\") " pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:22.524454 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:22.524409    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32d95902-6260-46cb-abb8-d103e438b80b-proxy-tls\") pod \"sequence-graph-887e9-57cbf8598b-fvfbt\" (UID: \"32d95902-6260-46cb-abb8-d103e438b80b\") " pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:22.526916 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:22.526890    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32d95902-6260-46cb-abb8-d103e438b80b-proxy-tls\") pod \"sequence-graph-887e9-57cbf8598b-fvfbt\" (UID: \"32d95902-6260-46cb-abb8-d103e438b80b\") " pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:22.782667 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:22.782582    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:23.054304 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:23.054277    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"]
Sep 17 18:55:23.056248 ip-10-0-139-5 kubenswrapper[2557]: W0917 18:55:23.056223    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d95902_6260_46cb_abb8_d103e438b80b.slice/crio-6aa9f91502d3cd2fbb93204438e5dee1836dd14bfb555e7e228e12dd71b0ffb8 WatchSource:0}: Error finding container 6aa9f91502d3cd2fbb93204438e5dee1836dd14bfb555e7e228e12dd71b0ffb8: Status 404 returned error can't find the container with id 6aa9f91502d3cd2fbb93204438e5dee1836dd14bfb555e7e228e12dd71b0ffb8
Sep 17 18:55:23.592550 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:23.592518    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" event={"ID":"32d95902-6260-46cb-abb8-d103e438b80b","Type":"ContainerStarted","Data":"51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff"}
Sep 17 18:55:23.592737 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:23.592571    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" event={"ID":"32d95902-6260-46cb-abb8-d103e438b80b","Type":"ContainerStarted","Data":"6aa9f91502d3cd2fbb93204438e5dee1836dd14bfb555e7e228e12dd71b0ffb8"}
Sep 17 18:55:23.592737 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:23.592609    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:55:23.612823 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:23.612771    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" podStartSLOduration=2.612755571 podStartE2EDuration="2.612755571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:23.611846676 +0000 UTC m=+996.967105448" watchObservedRunningTime="2026-09-17 18:55:23.612755571 +0000 UTC m=+996.968014343"
Sep 17 18:55:29.602405 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:55:29.602376    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 18:58:48.768685 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:58:48.768657    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:58:48.769702 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:58:48.769680    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 18:58:48.772003 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:58:48.771983    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 18:58:48.772800 ip-10-0-139-5 kubenswrapper[2557]: I0917 18:58:48.772783    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:02:50.165960 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.165881    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"]
Sep 17 19:02:50.168437 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.166145    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f" containerID="cri-o://ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa" gracePeriod=30
Sep 17 19:02:50.340608 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.340577    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"]
Sep 17 19:02:50.340873 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.340843    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container" containerID="cri-o://83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632" gracePeriod=30
Sep 17 19:02:50.341017 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.340902    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kube-rbac-proxy" containerID="cri-o://5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a" gracePeriod=30
Sep 17 19:02:50.497856 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.497817    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"]
Sep 17 19:02:50.501175 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.501159    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.503853 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.503830    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f962-predictor-serving-cert\""
Sep 17 19:02:50.503853 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.503848    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f962-kube-rbac-proxy-sar-config\""
Sep 17 19:02:50.522793 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.522767    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"]
Sep 17 19:02:50.580717 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.580685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxv8x\" (UniqueName: \"kubernetes.io/projected/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-api-access-cxv8x\") pod \"success-200-isvc-4f962-predictor-8576b94ccd-8d4zs\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") " pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.580872 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.580730    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9d1dae-fff4-4047-8847-95791de0b5ce-proxy-tls\") pod \"success-200-isvc-4f962-predictor-8576b94ccd-8d4zs\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") " pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.580872 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.580810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f962-predictor-8576b94ccd-8d4zs\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") " pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.681661 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.681618    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f962-predictor-8576b94ccd-8d4zs\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") " pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.681887 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.681765    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxv8x\" (UniqueName: \"kubernetes.io/projected/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-api-access-cxv8x\") pod \"success-200-isvc-4f962-predictor-8576b94ccd-8d4zs\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") " pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.681887 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.681793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9d1dae-fff4-4047-8847-95791de0b5ce-proxy-tls\") pod \"success-200-isvc-4f962-predictor-8576b94ccd-8d4zs\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") " pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.682324 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.682305    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f962-predictor-8576b94ccd-8d4zs\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") " pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.684152 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.684128    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9d1dae-fff4-4047-8847-95791de0b5ce-proxy-tls\") pod \"success-200-isvc-4f962-predictor-8576b94ccd-8d4zs\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") " pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.696714 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.696687    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxv8x\" (UniqueName: \"kubernetes.io/projected/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-api-access-cxv8x\") pod \"success-200-isvc-4f962-predictor-8576b94ccd-8d4zs\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") " pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:50.811878 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:50.811792    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:51.088580 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.088551    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"]
Sep 17 19:02:51.090811 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:02:51.090786    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9d1dae_fff4_4047_8847_95791de0b5ce.slice/crio-1a0721b394d1973a23f3461678bd2c48f44a3211d416a5cb9ed0c46bea0cff7a WatchSource:0}: Error finding container 1a0721b394d1973a23f3461678bd2c48f44a3211d416a5cb9ed0c46bea0cff7a: Status 404 returned error can't find the container with id 1a0721b394d1973a23f3461678bd2c48f44a3211d416a5cb9ed0c46bea0cff7a
Sep 17 19:02:51.371018 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.370943    2557 generic.go:356] "Generic (PLEG): container finished" podID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerID="5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a" exitCode=2
Sep 17 19:02:51.371383 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.371015    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" event={"ID":"af3f40cd-af17-4bb0-9127-09346f9e1954","Type":"ContainerDied","Data":"5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a"}
Sep 17 19:02:51.372784 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.372763    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" event={"ID":"4d9d1dae-fff4-4047-8847-95791de0b5ce","Type":"ContainerStarted","Data":"cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9"}
Sep 17 19:02:51.372897 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.372805    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" event={"ID":"4d9d1dae-fff4-4047-8847-95791de0b5ce","Type":"ContainerStarted","Data":"930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090"}
Sep 17 19:02:51.372897 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.372829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" event={"ID":"4d9d1dae-fff4-4047-8847-95791de0b5ce","Type":"ContainerStarted","Data":"1a0721b394d1973a23f3461678bd2c48f44a3211d416a5cb9ed0c46bea0cff7a"}
Sep 17 19:02:51.373007 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.372990    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:51.373135 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.373121    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:51.374678 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.374653    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 19:02:51.396295 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:51.396256    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podStartSLOduration=1.396242966 podStartE2EDuration="1.396242966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:51.394385914 +0000 UTC m=+1444.749644710" watchObservedRunningTime="2026-09-17 19:02:51.396242966 +0000 UTC m=+1444.751501735"
Sep 17 19:02:52.376904 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:52.376869    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 19:02:53.170140 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.170098    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 19:02:53.176290 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.176260    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 19:02:53.414286 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.414252    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:02:53.582568 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.582542    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 19:02:53.603176 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.603147    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfx5z\" (UniqueName: \"kubernetes.io/projected/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-api-access-kfx5z\") pod \"af3f40cd-af17-4bb0-9127-09346f9e1954\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") "
Sep 17 19:02:53.603332 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.603195    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-rbac-proxy-sar-config\") pod \"af3f40cd-af17-4bb0-9127-09346f9e1954\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") "
Sep 17 19:02:53.603332 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.603228    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af3f40cd-af17-4bb0-9127-09346f9e1954-proxy-tls\") pod \"af3f40cd-af17-4bb0-9127-09346f9e1954\" (UID: \"af3f40cd-af17-4bb0-9127-09346f9e1954\") "
Sep 17 19:02:53.603623 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.603590    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af3f40cd-af17-4bb0-9127-09346f9e1954" (UID: "af3f40cd-af17-4bb0-9127-09346f9e1954"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:02:53.605272 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.605244    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3f40cd-af17-4bb0-9127-09346f9e1954-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af3f40cd-af17-4bb0-9127-09346f9e1954" (UID: "af3f40cd-af17-4bb0-9127-09346f9e1954"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:53.605415 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.605295    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-api-access-kfx5z" (OuterVolumeSpecName: "kube-api-access-kfx5z") pod "af3f40cd-af17-4bb0-9127-09346f9e1954" (UID: "af3f40cd-af17-4bb0-9127-09346f9e1954"). InnerVolumeSpecName "kube-api-access-kfx5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:53.704674 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.704577    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:02:53.704674 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.704613    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af3f40cd-af17-4bb0-9127-09346f9e1954-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:02:53.704674 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:53.704627    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfx5z\" (UniqueName: \"kubernetes.io/projected/af3f40cd-af17-4bb0-9127-09346f9e1954-kube-api-access-kfx5z\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:02:54.385816 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.385774    2557 generic.go:356] "Generic (PLEG): container finished" podID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerID="83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632" exitCode=0
Sep 17 19:02:54.385816 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.385820    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"
Sep 17 19:02:54.386027 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.385859    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" event={"ID":"af3f40cd-af17-4bb0-9127-09346f9e1954","Type":"ContainerDied","Data":"83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632"}
Sep 17 19:02:54.386027 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.385924    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz" event={"ID":"af3f40cd-af17-4bb0-9127-09346f9e1954","Type":"ContainerDied","Data":"39d11fc975fcd6c1cd271e72cbd3194bec88883b3731ae6d909adf5d724e14cd"}
Sep 17 19:02:54.386027 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.385954    2557 scope.go:117] "RemoveContainer" containerID="5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a"
Sep 17 19:02:54.394528 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.394511    2557 scope.go:117] "RemoveContainer" containerID="83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632"
Sep 17 19:02:54.402066 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.402047    2557 scope.go:117] "RemoveContainer" containerID="5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a"
Sep 17 19:02:54.402349 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:02:54.402329    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a\": container with ID starting with 5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a not found: ID does not exist" containerID="5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a"
Sep 17 19:02:54.402431 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.402361    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a"} err="failed to get container status \"5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a\": rpc error: code = NotFound desc = could not find container \"5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a\": container with ID starting with 5971d753629fda226421cc41cb198325817f79c7a5a39f4fb2459842885bf93a not found: ID does not exist"
Sep 17 19:02:54.402431 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.402385    2557 scope.go:117] "RemoveContainer" containerID="83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632"
Sep 17 19:02:54.402623 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:02:54.402604    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632\": container with ID starting with 83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632 not found: ID does not exist" containerID="83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632"
Sep 17 19:02:54.402664 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.402629    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632"} err="failed to get container status \"83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632\": rpc error: code = NotFound desc = could not find container \"83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632\": container with ID starting with 83ac642f3e4ea2d7338971bb05cd6505d1bf78593281850512f18ef6aec66632 not found: ID does not exist"
Sep 17 19:02:54.408773 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.408747    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"]
Sep 17 19:02:54.413149 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.413123    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7989f-predictor-749fdfd4db-6qthz"]
Sep 17 19:02:54.835673 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:54.835643    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" path="/var/lib/kubelet/pods/af3f40cd-af17-4bb0-9127-09346f9e1954/volumes"
Sep 17 19:02:57.381315 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:57.381283    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:02:57.381786 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:57.381759    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 19:02:58.413422 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:02:58.413381    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:03:03.413431 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:03.413385    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:03:03.413832 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:03.413502    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 19:03:07.381755 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:07.381719    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 19:03:08.413365 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:08.413321    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:03:13.414139 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:13.414091    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:03:17.381880 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:17.381839    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 19:03:18.413964 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:18.413927    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:03:20.311960 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.311934    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 19:03:20.416698 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.416660    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3eac444-40fd-4d8e-b383-c1db59ddc96f-proxy-tls\") pod \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\" (UID: \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\") "
Sep 17 19:03:20.416869 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.416712    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3eac444-40fd-4d8e-b383-c1db59ddc96f-openshift-service-ca-bundle\") pod \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\" (UID: \"f3eac444-40fd-4d8e-b383-c1db59ddc96f\") "
Sep 17 19:03:20.417075 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.417053    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3eac444-40fd-4d8e-b383-c1db59ddc96f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f3eac444-40fd-4d8e-b383-c1db59ddc96f" (UID: "f3eac444-40fd-4d8e-b383-c1db59ddc96f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:20.418734 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.418704    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3eac444-40fd-4d8e-b383-c1db59ddc96f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3eac444-40fd-4d8e-b383-c1db59ddc96f" (UID: "f3eac444-40fd-4d8e-b383-c1db59ddc96f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:20.488659 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.488629    2557 generic.go:356] "Generic (PLEG): container finished" podID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerID="ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa" exitCode=0
Sep 17 19:03:20.488801 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.488676    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"
Sep 17 19:03:20.488801 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.488688    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" event={"ID":"f3eac444-40fd-4d8e-b383-c1db59ddc96f","Type":"ContainerDied","Data":"ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa"}
Sep 17 19:03:20.488801 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.488728    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958" event={"ID":"f3eac444-40fd-4d8e-b383-c1db59ddc96f","Type":"ContainerDied","Data":"d9d617e6353ab5d7f2f02401256b2a8021823f00f009aa14db65b6e5a8004014"}
Sep 17 19:03:20.488801 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.488749    2557 scope.go:117] "RemoveContainer" containerID="ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa"
Sep 17 19:03:20.497413 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.497398    2557 scope.go:117] "RemoveContainer" containerID="ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa"
Sep 17 19:03:20.497664 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:03:20.497647    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa\": container with ID starting with ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa not found: ID does not exist" containerID="ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa"
Sep 17 19:03:20.497707 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.497671    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa"} err="failed to get container status \"ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa\": rpc error: code = NotFound desc = could not find container \"ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa\": container with ID starting with ee31f7ab9fd1b087926027ccc85ca9e8a8e7414a52ab369d210f9bc0dd95befa not found: ID does not exist"
Sep 17 19:03:20.510840 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.510814    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"]
Sep 17 19:03:20.516495 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.516474    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7989f-64d5897b47-4j958"]
Sep 17 19:03:20.517146 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.517128    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3eac444-40fd-4d8e-b383-c1db59ddc96f-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:03:20.517202 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.517150    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3eac444-40fd-4d8e-b383-c1db59ddc96f-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:03:20.835080 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:20.835049    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" path="/var/lib/kubelet/pods/f3eac444-40fd-4d8e-b383-c1db59ddc96f/volumes"
Sep 17 19:03:27.381870 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:27.381828    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 19:03:36.318145 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.318109    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"]
Sep 17 19:03:36.318620 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.318379    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9" containerID="cri-o://51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff" gracePeriod=30
Sep 17 19:03:36.472601 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.472571    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"]
Sep 17 19:03:36.472844 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.472806    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container" containerID="cri-o://851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219" gracePeriod=30
Sep 17 19:03:36.472930 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.472887    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kube-rbac-proxy" containerID="cri-o://9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff" gracePeriod=30
Sep 17 19:03:36.671809 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.671778    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"]
Sep 17 19:03:36.672088 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.672077    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container"
Sep 17 19:03:36.672132 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.672090    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container"
Sep 17 19:03:36.672132 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.672114    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kube-rbac-proxy"
Sep 17 19:03:36.672132 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.672119    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kube-rbac-proxy"
Sep 17 19:03:36.672132 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.672129    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f"
Sep 17 19:03:36.672307 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.672135    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f"
Sep 17 19:03:36.672307 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.672180    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kserve-container"
Sep 17 19:03:36.672307 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.672187    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3eac444-40fd-4d8e-b383-c1db59ddc96f" containerName="switch-graph-7989f"
Sep 17 19:03:36.672307 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.672194    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3f40cd-af17-4bb0-9127-09346f9e1954" containerName="kube-rbac-proxy"
Sep 17 19:03:36.675249 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.675234    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.678743 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.678719    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5af63-kube-rbac-proxy-sar-config\""
Sep 17 19:03:36.678863 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.678717    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5af63-predictor-serving-cert\""
Sep 17 19:03:36.691460 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.691439    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"]
Sep 17 19:03:36.852511 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.852423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ffb887f-c274-4c74-bda0-1736984da495-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5af63-predictor-5945f9cd94-h6gbc\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") " pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.852511 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.852472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ffb887f-c274-4c74-bda0-1736984da495-proxy-tls\") pod \"success-200-isvc-5af63-predictor-5945f9cd94-h6gbc\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") " pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.852708 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.852553    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjqjv\" (UniqueName: \"kubernetes.io/projected/3ffb887f-c274-4c74-bda0-1736984da495-kube-api-access-xjqjv\") pod \"success-200-isvc-5af63-predictor-5945f9cd94-h6gbc\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") " pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.953757 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.953721    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjqjv\" (UniqueName: \"kubernetes.io/projected/3ffb887f-c274-4c74-bda0-1736984da495-kube-api-access-xjqjv\") pod \"success-200-isvc-5af63-predictor-5945f9cd94-h6gbc\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") " pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.953932 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.953776    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ffb887f-c274-4c74-bda0-1736984da495-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5af63-predictor-5945f9cd94-h6gbc\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") " pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.953932 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.953810    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ffb887f-c274-4c74-bda0-1736984da495-proxy-tls\") pod \"success-200-isvc-5af63-predictor-5945f9cd94-h6gbc\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") " pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.954560 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.954542    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ffb887f-c274-4c74-bda0-1736984da495-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5af63-predictor-5945f9cd94-h6gbc\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") " pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.956296 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.956279    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ffb887f-c274-4c74-bda0-1736984da495-proxy-tls\") pod \"success-200-isvc-5af63-predictor-5945f9cd94-h6gbc\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") " pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.964507 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.964485    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjqjv\" (UniqueName: \"kubernetes.io/projected/3ffb887f-c274-4c74-bda0-1736984da495-kube-api-access-xjqjv\") pod \"success-200-isvc-5af63-predictor-5945f9cd94-h6gbc\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") " pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:36.985566 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:36.985534    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:37.262344 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:37.262227    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"]
Sep 17 19:03:37.266778 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:03:37.266754    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffb887f_c274_4c74_bda0_1736984da495.slice/crio-349aca2ebb4a8163a2419693d8bc47b114cdb37646c1dd5b71a4370c86566f3a WatchSource:0}: Error finding container 349aca2ebb4a8163a2419693d8bc47b114cdb37646c1dd5b71a4370c86566f3a: Status 404 returned error can't find the container with id 349aca2ebb4a8163a2419693d8bc47b114cdb37646c1dd5b71a4370c86566f3a
Sep 17 19:03:37.382985 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:37.382961    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:03:37.559030 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:37.558952    2557 generic.go:356] "Generic (PLEG): container finished" podID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerID="9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff" exitCode=2
Sep 17 19:03:37.559030 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:37.559023    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" event={"ID":"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0","Type":"ContainerDied","Data":"9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff"}
Sep 17 19:03:37.560874 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:37.560852    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" event={"ID":"3ffb887f-c274-4c74-bda0-1736984da495","Type":"ContainerStarted","Data":"248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964"}
Sep 17 19:03:37.561008 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:37.560901    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" event={"ID":"3ffb887f-c274-4c74-bda0-1736984da495","Type":"ContainerStarted","Data":"a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8"}
Sep 17 19:03:37.561008 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:37.560932    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" event={"ID":"3ffb887f-c274-4c74-bda0-1736984da495","Type":"ContainerStarted","Data":"349aca2ebb4a8163a2419693d8bc47b114cdb37646c1dd5b71a4370c86566f3a"}
Sep 17 19:03:37.561104 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:37.561009    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:38.565497 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:38.565457    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:38.566996 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:38.566968    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 19:03:39.355097 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.355051    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 19:03:39.360489 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.360460    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 19:03:39.569979 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.569942    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 19:03:39.600410 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.600379    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:03:39.830166 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.830141    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 19:03:39.853899 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.853844    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podStartSLOduration=3.85382793 podStartE2EDuration="3.85382793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:37.595906848 +0000 UTC m=+1490.951165641" watchObservedRunningTime="2026-09-17 19:03:39.85382793 +0000 UTC m=+1493.209086702"
Sep 17 19:03:39.978568 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.978471    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wg7w\" (UniqueName: \"kubernetes.io/projected/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-api-access-5wg7w\") pod \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") "
Sep 17 19:03:39.978568 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.978541    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-proxy-tls\") pod \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") "
Sep 17 19:03:39.978789 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.978758    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-rbac-proxy-sar-config\") pod \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\" (UID: \"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0\") "
Sep 17 19:03:39.979096 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.979070    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" (UID: "29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:39.980632 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.980611    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-api-access-5wg7w" (OuterVolumeSpecName: "kube-api-access-5wg7w") pod "29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" (UID: "29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0"). InnerVolumeSpecName "kube-api-access-5wg7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:39.980688 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:39.980612    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" (UID: "29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:40.079934 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.079892    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:03:40.079934 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.079930    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:03:40.079934 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.079944    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wg7w\" (UniqueName: \"kubernetes.io/projected/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0-kube-api-access-5wg7w\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:03:40.575939 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.575905    2557 generic.go:356] "Generic (PLEG): container finished" podID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerID="851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219" exitCode=0
Sep 17 19:03:40.576449 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.575947    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" event={"ID":"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0","Type":"ContainerDied","Data":"851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219"}
Sep 17 19:03:40.576449 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.575960    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"
Sep 17 19:03:40.576449 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.575998    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl" event={"ID":"29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0","Type":"ContainerDied","Data":"14a0a63dfd2525c09671bf3c2c97d0e3e274de2afe4e84832f56e86e3ba1ca9c"}
Sep 17 19:03:40.576449 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.576030    2557 scope.go:117] "RemoveContainer" containerID="9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff"
Sep 17 19:03:40.584649 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.584633    2557 scope.go:117] "RemoveContainer" containerID="851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219"
Sep 17 19:03:40.592158 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.592138    2557 scope.go:117] "RemoveContainer" containerID="9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff"
Sep 17 19:03:40.592480 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:03:40.592461    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff\": container with ID starting with 9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff not found: ID does not exist" containerID="9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff"
Sep 17 19:03:40.592534 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.592487    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff"} err="failed to get container status \"9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff\": rpc error: code = NotFound desc = could not find container \"9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff\": container with ID starting with 9816ca2d329ae7149382531cb94fb7b65799741fc9f8b4cd5987dbcd26eaf2ff not found: ID does not exist"
Sep 17 19:03:40.592534 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.592503    2557 scope.go:117] "RemoveContainer" containerID="851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219"
Sep 17 19:03:40.592730 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:03:40.592709    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219\": container with ID starting with 851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219 not found: ID does not exist" containerID="851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219"
Sep 17 19:03:40.592788 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.592739    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219"} err="failed to get container status \"851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219\": rpc error: code = NotFound desc = could not find container \"851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219\": container with ID starting with 851a22a1f29fbb1bc299153a6ed83e64ee04768a46aadc563b979a70f1049219 not found: ID does not exist"
Sep 17 19:03:40.598126 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.598104    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"]
Sep 17 19:03:40.602605 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.602578    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-887e9-predictor-7989ccbd74-bm7jl"]
Sep 17 19:03:40.835264 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:40.835174    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" path="/var/lib/kubelet/pods/29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0/volumes"
Sep 17 19:03:44.574657 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:44.574624    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:03:44.575187 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:44.575161    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 19:03:44.599873 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:44.599841    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:03:48.793832 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:48.793799    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:03:48.794819 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:48.794795    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:03:48.797233 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:48.797198    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:03:48.798044 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:48.798028    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:03:49.600394 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:49.600351    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:03:49.600583 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:49.600460    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 19:03:50.471941 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.471911    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"]
Sep 17 19:03:50.472336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.472230    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container"
Sep 17 19:03:50.472336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.472242    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container"
Sep 17 19:03:50.472336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.472257    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kube-rbac-proxy"
Sep 17 19:03:50.472336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.472263    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kube-rbac-proxy"
Sep 17 19:03:50.472336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.472314    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kserve-container"
Sep 17 19:03:50.472336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.472324    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="29491bc1-33cb-41d4-8d5a-6b94f5d4f8d0" containerName="kube-rbac-proxy"
Sep 17 19:03:50.476797 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.476783    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:50.479669 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.479645    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-4f962-serving-cert\""
Sep 17 19:03:50.479669 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.479659    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-4f962-kube-rbac-proxy-sar-config\""
Sep 17 19:03:50.492586 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.492563    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"]
Sep 17 19:03:50.559746 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.559708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e735052-5aaf-4b0d-ac7e-7377d1fced33-openshift-service-ca-bundle\") pod \"ensemble-graph-4f962-798db48b9d-tcld9\" (UID: \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\") " pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:50.559910 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.559760    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e735052-5aaf-4b0d-ac7e-7377d1fced33-proxy-tls\") pod \"ensemble-graph-4f962-798db48b9d-tcld9\" (UID: \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\") " pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:50.660478 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.660447    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e735052-5aaf-4b0d-ac7e-7377d1fced33-openshift-service-ca-bundle\") pod \"ensemble-graph-4f962-798db48b9d-tcld9\" (UID: \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\") " pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:50.660646 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.660495    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e735052-5aaf-4b0d-ac7e-7377d1fced33-proxy-tls\") pod \"ensemble-graph-4f962-798db48b9d-tcld9\" (UID: \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\") " pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:50.660646 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:03:50.660613    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-4f962-serving-cert: secret "ensemble-graph-4f962-serving-cert" not found
Sep 17 19:03:50.660724 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:03:50.660667    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e735052-5aaf-4b0d-ac7e-7377d1fced33-proxy-tls podName:6e735052-5aaf-4b0d-ac7e-7377d1fced33 nodeName:}" failed. No retries permitted until 2026-09-17 19:03:51.160650664 +0000 UTC m=+1504.515909414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e735052-5aaf-4b0d-ac7e-7377d1fced33-proxy-tls") pod "ensemble-graph-4f962-798db48b9d-tcld9" (UID: "6e735052-5aaf-4b0d-ac7e-7377d1fced33") : secret "ensemble-graph-4f962-serving-cert" not found
Sep 17 19:03:50.661057 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:50.661036    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e735052-5aaf-4b0d-ac7e-7377d1fced33-openshift-service-ca-bundle\") pod \"ensemble-graph-4f962-798db48b9d-tcld9\" (UID: \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\") " pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:51.165980 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:51.165941    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e735052-5aaf-4b0d-ac7e-7377d1fced33-proxy-tls\") pod \"ensemble-graph-4f962-798db48b9d-tcld9\" (UID: \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\") " pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:51.168461 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:51.168430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e735052-5aaf-4b0d-ac7e-7377d1fced33-proxy-tls\") pod \"ensemble-graph-4f962-798db48b9d-tcld9\" (UID: \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\") " pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:51.386914 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:51.386877    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:51.659307 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:51.659283    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"]
Sep 17 19:03:51.661971 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:03:51.661949    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e735052_5aaf_4b0d_ac7e_7377d1fced33.slice/crio-984b14cb622c243e5d66dc722628c24810c8e5a59db33c7bfd2b67f0ae0380d3 WatchSource:0}: Error finding container 984b14cb622c243e5d66dc722628c24810c8e5a59db33c7bfd2b67f0ae0380d3: Status 404 returned error can't find the container with id 984b14cb622c243e5d66dc722628c24810c8e5a59db33c7bfd2b67f0ae0380d3
Sep 17 19:03:52.628851 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:52.628811    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" event={"ID":"6e735052-5aaf-4b0d-ac7e-7377d1fced33","Type":"ContainerStarted","Data":"1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4"}
Sep 17 19:03:52.629042 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:52.628867    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" event={"ID":"6e735052-5aaf-4b0d-ac7e-7377d1fced33","Type":"ContainerStarted","Data":"984b14cb622c243e5d66dc722628c24810c8e5a59db33c7bfd2b67f0ae0380d3"}
Sep 17 19:03:52.629042 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:52.628956    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:52.663778 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:52.663730    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" podStartSLOduration=2.663715408 podStartE2EDuration="2.663715408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:52.662023037 +0000 UTC m=+1506.017281821" watchObservedRunningTime="2026-09-17 19:03:52.663715408 +0000 UTC m=+1506.018974181"
Sep 17 19:03:54.575901 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:54.575855    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 19:03:54.600286 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:54.600251    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:03:58.638727 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:58.638696    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:03:59.600752 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:03:59.600713    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:00.640751 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:00.640721    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"]
Sep 17 19:04:00.641118 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:00.640942    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container" containerID="cri-o://930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090" gracePeriod=30
Sep 17 19:04:00.641118 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:00.640970    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kube-rbac-proxy" containerID="cri-o://cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9" gracePeriod=30
Sep 17 19:04:01.322976 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:01.322946    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"]
Sep 17 19:04:01.323148 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:01.323114    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962" containerID="cri-o://1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4" gracePeriod=30
Sep 17 19:04:01.665743 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:01.665665    2557 generic.go:356] "Generic (PLEG): container finished" podID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerID="cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9" exitCode=2
Sep 17 19:04:01.665743 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:01.665737    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" event={"ID":"4d9d1dae-fff4-4047-8847-95791de0b5ce","Type":"ContainerDied","Data":"cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9"}
Sep 17 19:04:02.377858 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:02.377818    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 19:04:03.296274 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.296230    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"]
Sep 17 19:04:03.299642 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.299626    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:03.304969 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:03.304940    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"success-200-isvc-bd292-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-139-5.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-139-5.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bd292-predictor-serving-cert\"" type="*v1.Secret"
Sep 17 19:04:03.329529 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:03.329478    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"success-200-isvc-bd292-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-139-5.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-139-5.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bd292-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 17 19:04:03.329772 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:03.329747    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" is forbidden: User \"system:node:ip-10-0-139-5.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-139-5.ec2.internal' and this object" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:03.360388 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.360354    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269ba13f-3f67-453c-a0f2-5300a3703170-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") " pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:03.360552 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.360404    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9bbp\" (UniqueName: \"kubernetes.io/projected/269ba13f-3f67-453c-a0f2-5300a3703170-kube-api-access-m9bbp\") pod \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") " pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:03.360552 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.360469    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269ba13f-3f67-453c-a0f2-5300a3703170-proxy-tls\") pod \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") " pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:03.409140 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.409107    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"]
Sep 17 19:04:03.461535 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.461495    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269ba13f-3f67-453c-a0f2-5300a3703170-proxy-tls\") pod \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") " pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:03.461701 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.461559    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269ba13f-3f67-453c-a0f2-5300a3703170-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") " pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:03.461701 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.461588    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9bbp\" (UniqueName: \"kubernetes.io/projected/269ba13f-3f67-453c-a0f2-5300a3703170-kube-api-access-m9bbp\") pod \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") " pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:03.473601 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.473571    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9bbp\" (UniqueName: \"kubernetes.io/projected/269ba13f-3f67-453c-a0f2-5300a3703170-kube-api-access-m9bbp\") pod \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") " pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:03.637373 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.637335    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:03.884579 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.884557    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:04:03.965442 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.965350    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxv8x\" (UniqueName: \"kubernetes.io/projected/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-api-access-cxv8x\") pod \"4d9d1dae-fff4-4047-8847-95791de0b5ce\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") "
Sep 17 19:04:03.965442 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.965401    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-rbac-proxy-sar-config\") pod \"4d9d1dae-fff4-4047-8847-95791de0b5ce\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") "
Sep 17 19:04:03.965656 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.965457    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9d1dae-fff4-4047-8847-95791de0b5ce-proxy-tls\") pod \"4d9d1dae-fff4-4047-8847-95791de0b5ce\" (UID: \"4d9d1dae-fff4-4047-8847-95791de0b5ce\") "
Sep 17 19:04:03.965861 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.965826    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d9d1dae-fff4-4047-8847-95791de0b5ce" (UID: "4d9d1dae-fff4-4047-8847-95791de0b5ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:03.967697 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.967666    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9d1dae-fff4-4047-8847-95791de0b5ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d9d1dae-fff4-4047-8847-95791de0b5ce" (UID: "4d9d1dae-fff4-4047-8847-95791de0b5ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:03.967798 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:03.967708    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-api-access-cxv8x" (OuterVolumeSpecName: "kube-api-access-cxv8x") pod "4d9d1dae-fff4-4047-8847-95791de0b5ce" (UID: "4d9d1dae-fff4-4047-8847-95791de0b5ce"). InnerVolumeSpecName "kube-api-access-cxv8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:04.066739 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.066702    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxv8x\" (UniqueName: \"kubernetes.io/projected/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-api-access-cxv8x\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:04.066739 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.066732    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d9d1dae-fff4-4047-8847-95791de0b5ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:04.066739 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.066742    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d9d1dae-fff4-4047-8847-95791de0b5ce-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:04.240089 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.240014    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bd292-predictor-serving-cert\""
Sep 17 19:04:04.244383 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.244366    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269ba13f-3f67-453c-a0f2-5300a3703170-proxy-tls\") pod \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") " pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:04.404391 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.404358    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bd292-kube-rbac-proxy-sar-config\""
Sep 17 19:04:04.412705 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.412680    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269ba13f-3f67-453c-a0f2-5300a3703170-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bd292-predictor-549cc54f5d-kc77q\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") " pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:04.509600 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.509509    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:04.575160 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.575120    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 19:04:04.600534 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.600502    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:04.679532 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.679500    2557 generic.go:356] "Generic (PLEG): container finished" podID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerID="930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090" exitCode=0
Sep 17 19:04:04.679703 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.679546    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" event={"ID":"4d9d1dae-fff4-4047-8847-95791de0b5ce","Type":"ContainerDied","Data":"930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090"}
Sep 17 19:04:04.679703 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.679589    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs" event={"ID":"4d9d1dae-fff4-4047-8847-95791de0b5ce","Type":"ContainerDied","Data":"1a0721b394d1973a23f3461678bd2c48f44a3211d416a5cb9ed0c46bea0cff7a"}
Sep 17 19:04:04.679703 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.679548    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"
Sep 17 19:04:04.679703 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.679613    2557 scope.go:117] "RemoveContainer" containerID="cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9"
Sep 17 19:04:04.688002 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.687981    2557 scope.go:117] "RemoveContainer" containerID="930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090"
Sep 17 19:04:04.695180 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.695163    2557 scope.go:117] "RemoveContainer" containerID="cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9"
Sep 17 19:04:04.695463 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:04.695442    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9\": container with ID starting with cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9 not found: ID does not exist" containerID="cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9"
Sep 17 19:04:04.695520 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.695470    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9"} err="failed to get container status \"cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9\": rpc error: code = NotFound desc = could not find container \"cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9\": container with ID starting with cc9c5220f655784044ccb3961edde41bf64a2100e01deeb51e8b78887d2e72e9 not found: ID does not exist"
Sep 17 19:04:04.695520 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.695487    2557 scope.go:117] "RemoveContainer" containerID="930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090"
Sep 17 19:04:04.695712 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:04.695696    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090\": container with ID starting with 930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090 not found: ID does not exist" containerID="930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090"
Sep 17 19:04:04.695756 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.695717    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090"} err="failed to get container status \"930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090\": rpc error: code = NotFound desc = could not find container \"930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090\": container with ID starting with 930cd45a413f11f2c5fde6e14bf8c36059d132a9eeaace082546691132259090 not found: ID does not exist"
Sep 17 19:04:04.713440 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.713419    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"]
Sep 17 19:04:04.722811 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.722789    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f962-predictor-8576b94ccd-8d4zs"]
Sep 17 19:04:04.804457 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.804373    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"]
Sep 17 19:04:04.808015 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:04:04.807991    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269ba13f_3f67_453c_a0f2_5300a3703170.slice/crio-df4aeb48de045463614fc91cb3fda022c5e08ecf38ebedaa23bf3ee25c6b4e7f WatchSource:0}: Error finding container df4aeb48de045463614fc91cb3fda022c5e08ecf38ebedaa23bf3ee25c6b4e7f: Status 404 returned error can't find the container with id df4aeb48de045463614fc91cb3fda022c5e08ecf38ebedaa23bf3ee25c6b4e7f
Sep 17 19:04:04.835380 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:04.835353    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" path="/var/lib/kubelet/pods/4d9d1dae-fff4-4047-8847-95791de0b5ce/volumes"
Sep 17 19:04:05.686649 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:05.686614    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" event={"ID":"269ba13f-3f67-453c-a0f2-5300a3703170","Type":"ContainerStarted","Data":"d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915"}
Sep 17 19:04:05.687033 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:05.686668    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" event={"ID":"269ba13f-3f67-453c-a0f2-5300a3703170","Type":"ContainerStarted","Data":"238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c"}
Sep 17 19:04:05.687033 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:05.686689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" event={"ID":"269ba13f-3f67-453c-a0f2-5300a3703170","Type":"ContainerStarted","Data":"df4aeb48de045463614fc91cb3fda022c5e08ecf38ebedaa23bf3ee25c6b4e7f"}
Sep 17 19:04:05.687033 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:05.686712    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:05.687033 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:05.686861    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:05.688349 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:05.688315    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:04:05.749877 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:05.749823    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podStartSLOduration=3.749803792 podStartE2EDuration="3.749803792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:05.748996292 +0000 UTC m=+1519.104255084" watchObservedRunningTime="2026-09-17 19:04:05.749803792 +0000 UTC m=+1519.105062565"
Sep 17 19:04:06.464954 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.464931    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 19:04:06.586471 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.586379    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d95902-6260-46cb-abb8-d103e438b80b-openshift-service-ca-bundle\") pod \"32d95902-6260-46cb-abb8-d103e438b80b\" (UID: \"32d95902-6260-46cb-abb8-d103e438b80b\") "
Sep 17 19:04:06.586471 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.586445    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32d95902-6260-46cb-abb8-d103e438b80b-proxy-tls\") pod \"32d95902-6260-46cb-abb8-d103e438b80b\" (UID: \"32d95902-6260-46cb-abb8-d103e438b80b\") "
Sep 17 19:04:06.586874 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.586769    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d95902-6260-46cb-abb8-d103e438b80b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "32d95902-6260-46cb-abb8-d103e438b80b" (UID: "32d95902-6260-46cb-abb8-d103e438b80b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:06.588577 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.588553    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d95902-6260-46cb-abb8-d103e438b80b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32d95902-6260-46cb-abb8-d103e438b80b" (UID: "32d95902-6260-46cb-abb8-d103e438b80b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:06.686966 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.686934    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d95902-6260-46cb-abb8-d103e438b80b-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:06.686966 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.686964    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32d95902-6260-46cb-abb8-d103e438b80b-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:06.691167 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.691134    2557 generic.go:356] "Generic (PLEG): container finished" podID="32d95902-6260-46cb-abb8-d103e438b80b" containerID="51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff" exitCode=0
Sep 17 19:04:06.691314 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.691180    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"
Sep 17 19:04:06.691314 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.691227    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" event={"ID":"32d95902-6260-46cb-abb8-d103e438b80b","Type":"ContainerDied","Data":"51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff"}
Sep 17 19:04:06.691314 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.691284    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt" event={"ID":"32d95902-6260-46cb-abb8-d103e438b80b","Type":"ContainerDied","Data":"6aa9f91502d3cd2fbb93204438e5dee1836dd14bfb555e7e228e12dd71b0ffb8"}
Sep 17 19:04:06.691314 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.691308    2557 scope.go:117] "RemoveContainer" containerID="51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff"
Sep 17 19:04:06.691910 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.691876    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:04:06.699921 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.699906    2557 scope.go:117] "RemoveContainer" containerID="51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff"
Sep 17 19:04:06.700181 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:06.700160    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff\": container with ID starting with 51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff not found: ID does not exist" containerID="51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff"
Sep 17 19:04:06.700282 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.700187    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff"} err="failed to get container status \"51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff\": rpc error: code = NotFound desc = could not find container \"51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff\": container with ID starting with 51f1706a4bc4b3d8113dc530bc714f66dddeb77b7021d29826fe56fac3a753ff not found: ID does not exist"
Sep 17 19:04:06.723564 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.723538    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"]
Sep 17 19:04:06.735808 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.735772    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-887e9-57cbf8598b-fvfbt"]
Sep 17 19:04:06.835506 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:06.835473    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d95902-6260-46cb-abb8-d103e438b80b" path="/var/lib/kubelet/pods/32d95902-6260-46cb-abb8-d103e438b80b/volumes"
Sep 17 19:04:08.637936 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:08.637893    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:11.696878 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:11.696853    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:11.697331 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:11.697306    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:04:13.637353 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:13.637313    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:13.637790 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:13.637441    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:04:14.576126 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:14.576085    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 19:04:18.636983 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:18.636941    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:21.698064 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:21.698021    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:04:23.637184 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:23.637146    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:24.576540 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:24.576510    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:04:28.637860 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:28.637822    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:31.459665 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.459643    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:04:31.589888 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.589782    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e735052-5aaf-4b0d-ac7e-7377d1fced33-openshift-service-ca-bundle\") pod \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\" (UID: \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\") "
Sep 17 19:04:31.589888 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.589838    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e735052-5aaf-4b0d-ac7e-7377d1fced33-proxy-tls\") pod \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\" (UID: \"6e735052-5aaf-4b0d-ac7e-7377d1fced33\") "
Sep 17 19:04:31.590169 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.590144    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e735052-5aaf-4b0d-ac7e-7377d1fced33-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6e735052-5aaf-4b0d-ac7e-7377d1fced33" (UID: "6e735052-5aaf-4b0d-ac7e-7377d1fced33"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:31.591892 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.591872    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e735052-5aaf-4b0d-ac7e-7377d1fced33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e735052-5aaf-4b0d-ac7e-7377d1fced33" (UID: "6e735052-5aaf-4b0d-ac7e-7377d1fced33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:31.690746 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.690707    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e735052-5aaf-4b0d-ac7e-7377d1fced33-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:31.690746 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.690739    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e735052-5aaf-4b0d-ac7e-7377d1fced33-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:31.698340 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.698306    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:04:31.793714 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.793679    2557 generic.go:356] "Generic (PLEG): container finished" podID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerID="1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4" exitCode=0
Sep 17 19:04:31.793883 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.793721    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"
Sep 17 19:04:31.793883 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.793754    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" event={"ID":"6e735052-5aaf-4b0d-ac7e-7377d1fced33","Type":"ContainerDied","Data":"1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4"}
Sep 17 19:04:31.793883 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.793797    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9" event={"ID":"6e735052-5aaf-4b0d-ac7e-7377d1fced33","Type":"ContainerDied","Data":"984b14cb622c243e5d66dc722628c24810c8e5a59db33c7bfd2b67f0ae0380d3"}
Sep 17 19:04:31.793883 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.793818    2557 scope.go:117] "RemoveContainer" containerID="1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4"
Sep 17 19:04:31.802699 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.802683    2557 scope.go:117] "RemoveContainer" containerID="1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4"
Sep 17 19:04:31.802952 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:31.802936    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4\": container with ID starting with 1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4 not found: ID does not exist" containerID="1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4"
Sep 17 19:04:31.803002 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.802959    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4"} err="failed to get container status \"1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4\": rpc error: code = NotFound desc = could not find container \"1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4\": container with ID starting with 1c454ce9b099ef59b1a49d84ab001256cac4568d6614b18d91aeb78f82f5f1c4 not found: ID does not exist"
Sep 17 19:04:31.822678 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.822651    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"]
Sep 17 19:04:31.829949 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:31.829928    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-4f962-798db48b9d-tcld9"]
Sep 17 19:04:32.835509 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:32.835477    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" path="/var/lib/kubelet/pods/6e735052-5aaf-4b0d-ac7e-7377d1fced33/volumes"
Sep 17 19:04:36.691093 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691062    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"]
Sep 17 19:04:36.691481 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691367    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kube-rbac-proxy"
Sep 17 19:04:36.691481 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691381    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kube-rbac-proxy"
Sep 17 19:04:36.691481 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691401    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962"
Sep 17 19:04:36.691481 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691407    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962"
Sep 17 19:04:36.691481 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691414    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9"
Sep 17 19:04:36.691481 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691419    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9"
Sep 17 19:04:36.691481 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691426    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container"
Sep 17 19:04:36.691481 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691431    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container"
Sep 17 19:04:36.691481 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691482    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d95902-6260-46cb-abb8-d103e438b80b" containerName="sequence-graph-887e9"
Sep 17 19:04:36.691760 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691490    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e735052-5aaf-4b0d-ac7e-7377d1fced33" containerName="ensemble-graph-4f962"
Sep 17 19:04:36.691760 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691496    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kube-rbac-proxy"
Sep 17 19:04:36.691760 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.691501    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9d1dae-fff4-4047-8847-95791de0b5ce" containerName="kserve-container"
Sep 17 19:04:36.695807 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.695788    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:36.698502 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.698483    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-5af63-kube-rbac-proxy-sar-config\""
Sep 17 19:04:36.698884 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.698869    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-5af63-serving-cert\""
Sep 17 19:04:36.728328 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.728298    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"]
Sep 17 19:04:36.835384 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.835352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56f57392-9980-45e5-acc4-57bf50834e34-proxy-tls\") pod \"sequence-graph-5af63-859f6674f8-mf5dr\" (UID: \"56f57392-9980-45e5-acc4-57bf50834e34\") " pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:36.835540 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.835436    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56f57392-9980-45e5-acc4-57bf50834e34-openshift-service-ca-bundle\") pod \"sequence-graph-5af63-859f6674f8-mf5dr\" (UID: \"56f57392-9980-45e5-acc4-57bf50834e34\") " pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:36.936513 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.936482    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56f57392-9980-45e5-acc4-57bf50834e34-proxy-tls\") pod \"sequence-graph-5af63-859f6674f8-mf5dr\" (UID: \"56f57392-9980-45e5-acc4-57bf50834e34\") " pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:36.936672 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.936534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56f57392-9980-45e5-acc4-57bf50834e34-openshift-service-ca-bundle\") pod \"sequence-graph-5af63-859f6674f8-mf5dr\" (UID: \"56f57392-9980-45e5-acc4-57bf50834e34\") " pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:36.937077 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.937056    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56f57392-9980-45e5-acc4-57bf50834e34-openshift-service-ca-bundle\") pod \"sequence-graph-5af63-859f6674f8-mf5dr\" (UID: \"56f57392-9980-45e5-acc4-57bf50834e34\") " pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:36.938885 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:36.938859    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56f57392-9980-45e5-acc4-57bf50834e34-proxy-tls\") pod \"sequence-graph-5af63-859f6674f8-mf5dr\" (UID: \"56f57392-9980-45e5-acc4-57bf50834e34\") " pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:37.004969 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:37.004888    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:37.276831 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:37.276806    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"]
Sep 17 19:04:37.279493 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:04:37.279465    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f57392_9980_45e5_acc4_57bf50834e34.slice/crio-307cb09ea501c3200ecda7ec1fea067f77612c150270ff570a931da3d0962587 WatchSource:0}: Error finding container 307cb09ea501c3200ecda7ec1fea067f77612c150270ff570a931da3d0962587: Status 404 returned error can't find the container with id 307cb09ea501c3200ecda7ec1fea067f77612c150270ff570a931da3d0962587
Sep 17 19:04:37.821101 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:37.821069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" event={"ID":"56f57392-9980-45e5-acc4-57bf50834e34","Type":"ContainerStarted","Data":"e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d"}
Sep 17 19:04:37.821516 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:37.821119    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" event={"ID":"56f57392-9980-45e5-acc4-57bf50834e34","Type":"ContainerStarted","Data":"307cb09ea501c3200ecda7ec1fea067f77612c150270ff570a931da3d0962587"}
Sep 17 19:04:37.821516 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:37.821186    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:37.842816 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:37.842769    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" podStartSLOduration=1.8427556489999999 podStartE2EDuration="1.842755649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:37.841295087 +0000 UTC m=+1551.196553874" watchObservedRunningTime="2026-09-17 19:04:37.842755649 +0000 UTC m=+1551.198014423"
Sep 17 19:04:41.698076 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:41.698038    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 19:04:43.830906 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:43.830875    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:04:46.680756 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.680720    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"]
Sep 17 19:04:46.681327 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.680968    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63" containerID="cri-o://e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d" gracePeriod=30
Sep 17 19:04:46.807391 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.807359    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"]
Sep 17 19:04:46.807641 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.807609    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container" containerID="cri-o://a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8" gracePeriod=30
Sep 17 19:04:46.807720 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.807646    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kube-rbac-proxy" containerID="cri-o://248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964" gracePeriod=30
Sep 17 19:04:46.928242 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.928180    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"]
Sep 17 19:04:46.931502 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.931448    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:46.934381 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.934361    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-058d2-predictor-serving-cert\""
Sep 17 19:04:46.934867 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.934849    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-058d2-kube-rbac-proxy-sar-config\""
Sep 17 19:04:46.950401 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:46.950379    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"]
Sep 17 19:04:47.015449 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.015421    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2fz\" (UniqueName: \"kubernetes.io/projected/569ab150-dba1-460c-b101-b7044e1cfb47-kube-api-access-nl2fz\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.015619 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.015471    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/569ab150-dba1-460c-b101-b7044e1cfb47-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.015619 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.015553    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/569ab150-dba1-460c-b101-b7044e1cfb47-proxy-tls\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.115971 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.115933    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2fz\" (UniqueName: \"kubernetes.io/projected/569ab150-dba1-460c-b101-b7044e1cfb47-kube-api-access-nl2fz\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.116157 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.115981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/569ab150-dba1-460c-b101-b7044e1cfb47-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.116157 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.116021    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/569ab150-dba1-460c-b101-b7044e1cfb47-proxy-tls\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.116157 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:47.116147    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-058d2-predictor-serving-cert: secret "success-200-isvc-058d2-predictor-serving-cert" not found
Sep 17 19:04:47.116356 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:47.116205    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569ab150-dba1-460c-b101-b7044e1cfb47-proxy-tls podName:569ab150-dba1-460c-b101-b7044e1cfb47 nodeName:}" failed. No retries permitted until 2026-09-17 19:04:47.616186968 +0000 UTC m=+1560.971445724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/569ab150-dba1-460c-b101-b7044e1cfb47-proxy-tls") pod "success-200-isvc-058d2-predictor-6df78f8b6-drsmk" (UID: "569ab150-dba1-460c-b101-b7044e1cfb47") : secret "success-200-isvc-058d2-predictor-serving-cert" not found
Sep 17 19:04:47.116741 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.116721    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/569ab150-dba1-460c-b101-b7044e1cfb47-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.139014 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.138981    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2fz\" (UniqueName: \"kubernetes.io/projected/569ab150-dba1-460c-b101-b7044e1cfb47-kube-api-access-nl2fz\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.620570 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.620530    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/569ab150-dba1-460c-b101-b7044e1cfb47-proxy-tls\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.622954 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.622933    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/569ab150-dba1-460c-b101-b7044e1cfb47-proxy-tls\") pod \"success-200-isvc-058d2-predictor-6df78f8b6-drsmk\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") " pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.841458 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.841422    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:47.870880 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.870800    2557 generic.go:356] "Generic (PLEG): container finished" podID="3ffb887f-c274-4c74-bda0-1736984da495" containerID="248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964" exitCode=2
Sep 17 19:04:47.870880 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:47.870849    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" event={"ID":"3ffb887f-c274-4c74-bda0-1736984da495","Type":"ContainerDied","Data":"248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964"}
Sep 17 19:04:48.124114 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:48.124067    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"]
Sep 17 19:04:48.126752 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:04:48.126726    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569ab150_dba1_460c_b101_b7044e1cfb47.slice/crio-a7c5c5818db0360f955acbff0dc672bbcf5ad53c4c4aead91571db4eba125190 WatchSource:0}: Error finding container a7c5c5818db0360f955acbff0dc672bbcf5ad53c4c4aead91571db4eba125190: Status 404 returned error can't find the container with id a7c5c5818db0360f955acbff0dc672bbcf5ad53c4c4aead91571db4eba125190
Sep 17 19:04:48.830369 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:48.830310    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:48.876225 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:48.876187    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" event={"ID":"569ab150-dba1-460c-b101-b7044e1cfb47","Type":"ContainerStarted","Data":"6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c"}
Sep 17 19:04:48.876626 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:48.876267    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" event={"ID":"569ab150-dba1-460c-b101-b7044e1cfb47","Type":"ContainerStarted","Data":"abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3"}
Sep 17 19:04:48.876626 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:48.876296    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:48.876626 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:48.876326    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:48.876626 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:48.876350    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" event={"ID":"569ab150-dba1-460c-b101-b7044e1cfb47","Type":"ContainerStarted","Data":"a7c5c5818db0360f955acbff0dc672bbcf5ad53c4c4aead91571db4eba125190"}
Sep 17 19:04:48.877845 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:48.877824    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:04:48.896658 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:48.896616    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podStartSLOduration=2.896600545 podStartE2EDuration="2.896600545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:48.896451166 +0000 UTC m=+1562.251709959" watchObservedRunningTime="2026-09-17 19:04:48.896600545 +0000 UTC m=+1562.251859317"
Sep 17 19:04:49.570700 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:49.570662    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 19:04:49.880266 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:49.880166    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:04:50.153004 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.152975    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:04:50.242101 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.242068    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjqjv\" (UniqueName: \"kubernetes.io/projected/3ffb887f-c274-4c74-bda0-1736984da495-kube-api-access-xjqjv\") pod \"3ffb887f-c274-4c74-bda0-1736984da495\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") "
Sep 17 19:04:50.242282 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.242134    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ffb887f-c274-4c74-bda0-1736984da495-kube-rbac-proxy-sar-config\") pod \"3ffb887f-c274-4c74-bda0-1736984da495\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") "
Sep 17 19:04:50.242282 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.242200    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ffb887f-c274-4c74-bda0-1736984da495-proxy-tls\") pod \"3ffb887f-c274-4c74-bda0-1736984da495\" (UID: \"3ffb887f-c274-4c74-bda0-1736984da495\") "
Sep 17 19:04:50.242528 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.242502    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffb887f-c274-4c74-bda0-1736984da495-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ffb887f-c274-4c74-bda0-1736984da495" (UID: "3ffb887f-c274-4c74-bda0-1736984da495"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:04:50.244229 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.244185    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ffb887f-c274-4c74-bda0-1736984da495-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ffb887f-c274-4c74-bda0-1736984da495" (UID: "3ffb887f-c274-4c74-bda0-1736984da495"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:50.244317 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.244249    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffb887f-c274-4c74-bda0-1736984da495-kube-api-access-xjqjv" (OuterVolumeSpecName: "kube-api-access-xjqjv") pod "3ffb887f-c274-4c74-bda0-1736984da495" (UID: "3ffb887f-c274-4c74-bda0-1736984da495"). InnerVolumeSpecName "kube-api-access-xjqjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:50.343437 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.343401    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ffb887f-c274-4c74-bda0-1736984da495-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:50.343437 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.343429    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjqjv\" (UniqueName: \"kubernetes.io/projected/3ffb887f-c274-4c74-bda0-1736984da495-kube-api-access-xjqjv\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:50.343437 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.343443    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ffb887f-c274-4c74-bda0-1736984da495-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:04:50.885837 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.885809    2557 generic.go:356] "Generic (PLEG): container finished" podID="3ffb887f-c274-4c74-bda0-1736984da495" containerID="a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8" exitCode=0
Sep 17 19:04:50.886336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.885871    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"
Sep 17 19:04:50.886336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.885895    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" event={"ID":"3ffb887f-c274-4c74-bda0-1736984da495","Type":"ContainerDied","Data":"a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8"}
Sep 17 19:04:50.886336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.885950    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc" event={"ID":"3ffb887f-c274-4c74-bda0-1736984da495","Type":"ContainerDied","Data":"349aca2ebb4a8163a2419693d8bc47b114cdb37646c1dd5b71a4370c86566f3a"}
Sep 17 19:04:50.886336 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.885976    2557 scope.go:117] "RemoveContainer" containerID="248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964"
Sep 17 19:04:50.894124 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.894108    2557 scope.go:117] "RemoveContainer" containerID="a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8"
Sep 17 19:04:50.901246 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.901228    2557 scope.go:117] "RemoveContainer" containerID="248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964"
Sep 17 19:04:50.901508 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:50.901491    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964\": container with ID starting with 248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964 not found: ID does not exist" containerID="248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964"
Sep 17 19:04:50.901568 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.901517    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964"} err="failed to get container status \"248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964\": rpc error: code = NotFound desc = could not find container \"248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964\": container with ID starting with 248e52de591aabcf3ef3c3c377b573e9a8ba4b57bf7d4e3a4da71a6608c5a964 not found: ID does not exist"
Sep 17 19:04:50.901568 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.901532    2557 scope.go:117] "RemoveContainer" containerID="a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8"
Sep 17 19:04:50.901768 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:04:50.901748    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8\": container with ID starting with a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8 not found: ID does not exist" containerID="a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8"
Sep 17 19:04:50.901808 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.901778    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8"} err="failed to get container status \"a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8\": rpc error: code = NotFound desc = could not find container \"a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8\": container with ID starting with a9d59c1cbf87d94d54d13e67e2892f861dd7d1879734059b856e19eba6cbfaa8 not found: ID does not exist"
Sep 17 19:04:50.906392 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.906370    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"]
Sep 17 19:04:50.911724 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:50.911705    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5af63-predictor-5945f9cd94-h6gbc"]
Sep 17 19:04:51.698899 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:51.698863    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:04:52.835512 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:52.835483    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ffb887f-c274-4c74-bda0-1736984da495" path="/var/lib/kubelet/pods/3ffb887f-c274-4c74-bda0-1736984da495/volumes"
Sep 17 19:04:53.830122 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:53.830076    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:54.885349 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:54.885314    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:04:54.885788 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:54.885736    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:04:58.830201 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:58.830153    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:04:58.834617 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:04:58.834594    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:05:03.829836 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:03.829797    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:05:04.886062 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:04.886016    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:05:08.829153 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:08.829114    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:05:10.872803 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.872770    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"]
Sep 17 19:05:10.873300 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.873121    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container"
Sep 17 19:05:10.873300 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.873134    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container"
Sep 17 19:05:10.873300 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.873152    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kube-rbac-proxy"
Sep 17 19:05:10.873300 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.873158    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kube-rbac-proxy"
Sep 17 19:05:10.873300 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.873226    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kube-rbac-proxy"
Sep 17 19:05:10.873300 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.873241    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ffb887f-c274-4c74-bda0-1736984da495" containerName="kserve-container"
Sep 17 19:05:10.876032 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.876017    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:10.881261 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.881240    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-bd292-serving-cert\""
Sep 17 19:05:10.881628 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.881609    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-bd292-kube-rbac-proxy-sar-config\""
Sep 17 19:05:10.894923 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.894897    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"]
Sep 17 19:05:10.905729 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.905705    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab9338b9-3a97-4139-aadb-6b22cf149616-openshift-service-ca-bundle\") pod \"ensemble-graph-bd292-8446b7b59d-5t2pc\" (UID: \"ab9338b9-3a97-4139-aadb-6b22cf149616\") " pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:10.905855 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:10.905790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9338b9-3a97-4139-aadb-6b22cf149616-proxy-tls\") pod \"ensemble-graph-bd292-8446b7b59d-5t2pc\" (UID: \"ab9338b9-3a97-4139-aadb-6b22cf149616\") " pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:11.010234 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.010171    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9338b9-3a97-4139-aadb-6b22cf149616-proxy-tls\") pod \"ensemble-graph-bd292-8446b7b59d-5t2pc\" (UID: \"ab9338b9-3a97-4139-aadb-6b22cf149616\") " pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:11.010421 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.010316    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab9338b9-3a97-4139-aadb-6b22cf149616-openshift-service-ca-bundle\") pod \"ensemble-graph-bd292-8446b7b59d-5t2pc\" (UID: \"ab9338b9-3a97-4139-aadb-6b22cf149616\") " pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:11.010999 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.010970    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab9338b9-3a97-4139-aadb-6b22cf149616-openshift-service-ca-bundle\") pod \"ensemble-graph-bd292-8446b7b59d-5t2pc\" (UID: \"ab9338b9-3a97-4139-aadb-6b22cf149616\") " pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:11.013354 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.013329    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9338b9-3a97-4139-aadb-6b22cf149616-proxy-tls\") pod \"ensemble-graph-bd292-8446b7b59d-5t2pc\" (UID: \"ab9338b9-3a97-4139-aadb-6b22cf149616\") " pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:11.185440 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.185349    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:11.462247 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.462158    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"]
Sep 17 19:05:11.464544 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:05:11.464513    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9338b9_3a97_4139_aadb_6b22cf149616.slice/crio-5455bd86a3a63edeb848018b4bf35a424e313b2bceeb2aff5ccb37909bbbc71b WatchSource:0}: Error finding container 5455bd86a3a63edeb848018b4bf35a424e313b2bceeb2aff5ccb37909bbbc71b: Status 404 returned error can't find the container with id 5455bd86a3a63edeb848018b4bf35a424e313b2bceeb2aff5ccb37909bbbc71b
Sep 17 19:05:11.972851 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.972814    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" event={"ID":"ab9338b9-3a97-4139-aadb-6b22cf149616","Type":"ContainerStarted","Data":"d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1"}
Sep 17 19:05:11.973262 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.972869    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" event={"ID":"ab9338b9-3a97-4139-aadb-6b22cf149616","Type":"ContainerStarted","Data":"5455bd86a3a63edeb848018b4bf35a424e313b2bceeb2aff5ccb37909bbbc71b"}
Sep 17 19:05:11.973262 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.972944    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:11.992139 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:11.992093    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" podStartSLOduration=1.992078083 podStartE2EDuration="1.992078083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:11.991139152 +0000 UTC m=+1585.346397936" watchObservedRunningTime="2026-09-17 19:05:11.992078083 +0000 UTC m=+1585.347336856"
Sep 17 19:05:13.829536 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:13.829500    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:05:14.886022 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:14.885988    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:05:16.855070 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.855047    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:05:16.958559 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.958482    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56f57392-9980-45e5-acc4-57bf50834e34-openshift-service-ca-bundle\") pod \"56f57392-9980-45e5-acc4-57bf50834e34\" (UID: \"56f57392-9980-45e5-acc4-57bf50834e34\") "
Sep 17 19:05:16.958559 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.958553    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56f57392-9980-45e5-acc4-57bf50834e34-proxy-tls\") pod \"56f57392-9980-45e5-acc4-57bf50834e34\" (UID: \"56f57392-9980-45e5-acc4-57bf50834e34\") "
Sep 17 19:05:16.958881 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.958861    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f57392-9980-45e5-acc4-57bf50834e34-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "56f57392-9980-45e5-acc4-57bf50834e34" (UID: "56f57392-9980-45e5-acc4-57bf50834e34"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:05:16.960697 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.960678    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f57392-9980-45e5-acc4-57bf50834e34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56f57392-9980-45e5-acc4-57bf50834e34" (UID: "56f57392-9980-45e5-acc4-57bf50834e34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:05:16.993997 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.993963    2557 generic.go:356] "Generic (PLEG): container finished" podID="56f57392-9980-45e5-acc4-57bf50834e34" containerID="e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d" exitCode=0
Sep 17 19:05:16.994127 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.994015    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"
Sep 17 19:05:16.994127 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.994047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" event={"ID":"56f57392-9980-45e5-acc4-57bf50834e34","Type":"ContainerDied","Data":"e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d"}
Sep 17 19:05:16.994127 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.994106    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr" event={"ID":"56f57392-9980-45e5-acc4-57bf50834e34","Type":"ContainerDied","Data":"307cb09ea501c3200ecda7ec1fea067f77612c150270ff570a931da3d0962587"}
Sep 17 19:05:16.994262 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:16.994129    2557 scope.go:117] "RemoveContainer" containerID="e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d"
Sep 17 19:05:17.002018 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:17.002002    2557 scope.go:117] "RemoveContainer" containerID="e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d"
Sep 17 19:05:17.002295 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:05:17.002272    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d\": container with ID starting with e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d not found: ID does not exist" containerID="e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d"
Sep 17 19:05:17.002391 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:17.002300    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d"} err="failed to get container status \"e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d\": rpc error: code = NotFound desc = could not find container \"e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d\": container with ID starting with e514ebf35073f21426db02b3793f602a8cfb0140ccc123b263c73d1e2f1bb55d not found: ID does not exist"
Sep 17 19:05:17.019133 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:17.019110    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"]
Sep 17 19:05:17.031417 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:17.031396    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5af63-859f6674f8-mf5dr"]
Sep 17 19:05:17.060144 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:17.060126    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56f57392-9980-45e5-acc4-57bf50834e34-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:05:17.060207 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:17.060151    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56f57392-9980-45e5-acc4-57bf50834e34-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:05:17.982107 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:17.982081    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:05:18.837483 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:18.837452    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56f57392-9980-45e5-acc4-57bf50834e34" path="/var/lib/kubelet/pods/56f57392-9980-45e5-acc4-57bf50834e34/volumes"
Sep 17 19:05:24.886788 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:24.886750    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:05:34.886447 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:34.886370    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:05:47.006873 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.006838    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"]
Sep 17 19:05:47.007305 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.007146    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63"
Sep 17 19:05:47.007305 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.007158    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63"
Sep 17 19:05:47.007305 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.007235    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="56f57392-9980-45e5-acc4-57bf50834e34" containerName="sequence-graph-5af63"
Sep 17 19:05:47.010132 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.010115    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:05:47.016110 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.016088    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-058d2-serving-cert\""
Sep 17 19:05:47.024403 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.024379    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-058d2-kube-rbac-proxy-sar-config\""
Sep 17 19:05:47.026778 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.026758    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"]
Sep 17 19:05:47.105253 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.105207    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3580520d-06dd-432e-8a6c-1f5f4d2f5996-proxy-tls\") pod \"sequence-graph-058d2-ff64f56f7-mmrgh\" (UID: \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\") " pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:05:47.105422 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.105331    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3580520d-06dd-432e-8a6c-1f5f4d2f5996-openshift-service-ca-bundle\") pod \"sequence-graph-058d2-ff64f56f7-mmrgh\" (UID: \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\") " pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:05:47.206072 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.206034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3580520d-06dd-432e-8a6c-1f5f4d2f5996-openshift-service-ca-bundle\") pod \"sequence-graph-058d2-ff64f56f7-mmrgh\" (UID: \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\") " pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:05:47.206269 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.206085    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3580520d-06dd-432e-8a6c-1f5f4d2f5996-proxy-tls\") pod \"sequence-graph-058d2-ff64f56f7-mmrgh\" (UID: \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\") " pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:05:47.206695 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.206674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3580520d-06dd-432e-8a6c-1f5f4d2f5996-openshift-service-ca-bundle\") pod \"sequence-graph-058d2-ff64f56f7-mmrgh\" (UID: \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\") " pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:05:47.208454 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.208433    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3580520d-06dd-432e-8a6c-1f5f4d2f5996-proxy-tls\") pod \"sequence-graph-058d2-ff64f56f7-mmrgh\" (UID: \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\") " pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:05:47.320687 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.320663    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:05:47.592150 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:47.592116    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"]
Sep 17 19:05:47.594910 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:05:47.594881    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3580520d_06dd_432e_8a6c_1f5f4d2f5996.slice/crio-3c81150e94d6f5921dd0665b7c9ada3aac8104f0aaa2d96679968700cf652315 WatchSource:0}: Error finding container 3c81150e94d6f5921dd0665b7c9ada3aac8104f0aaa2d96679968700cf652315: Status 404 returned error can't find the container with id 3c81150e94d6f5921dd0665b7c9ada3aac8104f0aaa2d96679968700cf652315
Sep 17 19:05:48.128756 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:48.128726    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" event={"ID":"3580520d-06dd-432e-8a6c-1f5f4d2f5996","Type":"ContainerStarted","Data":"6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781"}
Sep 17 19:05:48.129161 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:48.128777    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" event={"ID":"3580520d-06dd-432e-8a6c-1f5f4d2f5996","Type":"ContainerStarted","Data":"3c81150e94d6f5921dd0665b7c9ada3aac8104f0aaa2d96679968700cf652315"}
Sep 17 19:05:48.129161 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:48.128837    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:05:48.148169 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:48.148116    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" podStartSLOduration=2.1480999020000002 podStartE2EDuration="2.148099902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:48.147373259 +0000 UTC m=+1621.502675054" watchObservedRunningTime="2026-09-17 19:05:48.148099902 +0000 UTC m=+1621.503358674"
Sep 17 19:05:54.139042 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:05:54.139015    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:08:48.818244 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:08:48.818200    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:08:48.820539 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:08:48.820519    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:08:48.821309 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:08:48.821290    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:08:48.823924 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:08:48.823907    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:13:25.457168 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.457086    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"]
Sep 17 19:13:25.457728 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.457366    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292" containerID="cri-o://d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1" gracePeriod=30
Sep 17 19:13:25.588825 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.588793    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"]
Sep 17 19:13:25.589056 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.589009    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container" containerID="cri-o://238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c" gracePeriod=30
Sep 17 19:13:25.589148 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.589036    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kube-rbac-proxy" containerID="cri-o://d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915" gracePeriod=30
Sep 17 19:13:25.686147 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.686116    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"]
Sep 17 19:13:25.689544 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.689524    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.692250 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.692221    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3efeb-predictor-serving-cert\""
Sep 17 19:13:25.692392 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.692256    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3efeb-kube-rbac-proxy-sar-config\""
Sep 17 19:13:25.702129 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.702105    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"]
Sep 17 19:13:25.758392 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.758310    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-proxy-tls\") pod \"success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") " pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.758392 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.758359    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px788\" (UniqueName: \"kubernetes.io/projected/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-api-access-px788\") pod \"success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") " pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.758587 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.758423    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") " pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.859559 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.859523    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") " pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.859768 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.859595    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-proxy-tls\") pod \"success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") " pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.859768 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.859632    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px788\" (UniqueName: \"kubernetes.io/projected/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-api-access-px788\") pod \"success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") " pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.860238 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.860189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") " pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.862077 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.862056    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-proxy-tls\") pod \"success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") " pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.867865 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.867837    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px788\" (UniqueName: \"kubernetes.io/projected/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-api-access-px788\") pod \"success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") " pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:25.930393 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.930358    2557 generic.go:356] "Generic (PLEG): container finished" podID="269ba13f-3f67-453c-a0f2-5300a3703170" containerID="d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915" exitCode=2
Sep 17 19:13:25.930393 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:25.930398    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" event={"ID":"269ba13f-3f67-453c-a0f2-5300a3703170","Type":"ContainerDied","Data":"d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915"}
Sep 17 19:13:26.001306 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.001277    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:26.272015 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.271991    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"]
Sep 17 19:13:26.274111 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:13:26.274085    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c8ed24_e4e7_4809_a76a_6c49f54bbb1e.slice/crio-f00e26d2f5cc07667845676c99bf2953ace5db13f849f3806cab7b90cbbd3a33 WatchSource:0}: Error finding container f00e26d2f5cc07667845676c99bf2953ace5db13f849f3806cab7b90cbbd3a33: Status 404 returned error can't find the container with id f00e26d2f5cc07667845676c99bf2953ace5db13f849f3806cab7b90cbbd3a33
Sep 17 19:13:26.692403 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.692363    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 17 19:13:26.936791 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.936751    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" event={"ID":"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e","Type":"ContainerStarted","Data":"c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971"}
Sep 17 19:13:26.936960 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.936813    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" event={"ID":"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e","Type":"ContainerStarted","Data":"bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31"}
Sep 17 19:13:26.936960 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.936835    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" event={"ID":"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e","Type":"ContainerStarted","Data":"f00e26d2f5cc07667845676c99bf2953ace5db13f849f3806cab7b90cbbd3a33"}
Sep 17 19:13:26.936960 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.936865    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:26.937092 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.936976    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:26.938410 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.938384    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 19:13:26.955872 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:26.955783    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podStartSLOduration=1.955769075 podStartE2EDuration="1.955769075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:26.955108047 +0000 UTC m=+2080.310366832" watchObservedRunningTime="2026-09-17 19:13:26.955769075 +0000 UTC m=+2080.311027848"
Sep 17 19:13:27.941701 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:27.941661    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 19:13:27.981574 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:27.981539    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:13:28.941690 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.941666    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:13:28.946354 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.946323    2557 generic.go:356] "Generic (PLEG): container finished" podID="269ba13f-3f67-453c-a0f2-5300a3703170" containerID="238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c" exitCode=0
Sep 17 19:13:28.946676 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.946376    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"
Sep 17 19:13:28.946676 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.946407    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" event={"ID":"269ba13f-3f67-453c-a0f2-5300a3703170","Type":"ContainerDied","Data":"238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c"}
Sep 17 19:13:28.946676 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.946479    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q" event={"ID":"269ba13f-3f67-453c-a0f2-5300a3703170","Type":"ContainerDied","Data":"df4aeb48de045463614fc91cb3fda022c5e08ecf38ebedaa23bf3ee25c6b4e7f"}
Sep 17 19:13:28.946676 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.946508    2557 scope.go:117] "RemoveContainer" containerID="d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915"
Sep 17 19:13:28.954362 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.954347    2557 scope.go:117] "RemoveContainer" containerID="238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c"
Sep 17 19:13:28.961792 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.961735    2557 scope.go:117] "RemoveContainer" containerID="d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915"
Sep 17 19:13:28.962309 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:13:28.962283    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915\": container with ID starting with d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915 not found: ID does not exist" containerID="d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915"
Sep 17 19:13:28.962410 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.962317    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915"} err="failed to get container status \"d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915\": rpc error: code = NotFound desc = could not find container \"d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915\": container with ID starting with d013d76ca548fc35f288ff61c5e84bee68ffa04980769fa1650937ecfe988915 not found: ID does not exist"
Sep 17 19:13:28.962410 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.962335    2557 scope.go:117] "RemoveContainer" containerID="238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c"
Sep 17 19:13:28.962662 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:13:28.962628    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c\": container with ID starting with 238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c not found: ID does not exist" containerID="238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c"
Sep 17 19:13:28.962734 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:28.962667    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c"} err="failed to get container status \"238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c\": rpc error: code = NotFound desc = could not find container \"238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c\": container with ID starting with 238cbffeb9862c304265ed16b27b72798bdcb4b7fa10c1132041822ea483696c not found: ID does not exist"
Sep 17 19:13:29.086775 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.086745    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269ba13f-3f67-453c-a0f2-5300a3703170-kube-rbac-proxy-sar-config\") pod \"269ba13f-3f67-453c-a0f2-5300a3703170\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") "
Sep 17 19:13:29.086967 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.086782    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269ba13f-3f67-453c-a0f2-5300a3703170-proxy-tls\") pod \"269ba13f-3f67-453c-a0f2-5300a3703170\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") "
Sep 17 19:13:29.086967 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.086812    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9bbp\" (UniqueName: \"kubernetes.io/projected/269ba13f-3f67-453c-a0f2-5300a3703170-kube-api-access-m9bbp\") pod \"269ba13f-3f67-453c-a0f2-5300a3703170\" (UID: \"269ba13f-3f67-453c-a0f2-5300a3703170\") "
Sep 17 19:13:29.087131 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.087098    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269ba13f-3f67-453c-a0f2-5300a3703170-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "269ba13f-3f67-453c-a0f2-5300a3703170" (UID: "269ba13f-3f67-453c-a0f2-5300a3703170"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:29.088952 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.088930    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269ba13f-3f67-453c-a0f2-5300a3703170-kube-api-access-m9bbp" (OuterVolumeSpecName: "kube-api-access-m9bbp") pod "269ba13f-3f67-453c-a0f2-5300a3703170" (UID: "269ba13f-3f67-453c-a0f2-5300a3703170"). InnerVolumeSpecName "kube-api-access-m9bbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:29.089046 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.089027    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269ba13f-3f67-453c-a0f2-5300a3703170-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "269ba13f-3f67-453c-a0f2-5300a3703170" (UID: "269ba13f-3f67-453c-a0f2-5300a3703170"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:29.187824 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.187794    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269ba13f-3f67-453c-a0f2-5300a3703170-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:13:29.187824 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.187819    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269ba13f-3f67-453c-a0f2-5300a3703170-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:13:29.188019 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.187832    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9bbp\" (UniqueName: \"kubernetes.io/projected/269ba13f-3f67-453c-a0f2-5300a3703170-kube-api-access-m9bbp\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:13:29.272764 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.272730    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"]
Sep 17 19:13:29.277145 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:29.277118    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bd292-predictor-549cc54f5d-kc77q"]
Sep 17 19:13:30.835626 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:30.835595    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" path="/var/lib/kubelet/pods/269ba13f-3f67-453c-a0f2-5300a3703170/volumes"
Sep 17 19:13:32.946570 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:32.946538    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:13:32.947154 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:32.947129    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 19:13:32.980720 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:32.980676    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:13:37.980830 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:37.980792    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:13:37.981263 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:37.980903    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:13:42.947618 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:42.947579    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 19:13:42.980595 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:42.980561    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:13:47.981047 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:47.981003    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:13:48.843872 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:48.843844    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:13:48.847029 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:48.847005    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:13:48.847373 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:48.847347    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:13:48.850475 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:48.850458    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:13:52.947666 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:52.947619    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 19:13:52.980770 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:52.980738    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:13:55.601785 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:55.601762    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:13:55.602681 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:55.602666    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab9338b9-3a97-4139-aadb-6b22cf149616-openshift-service-ca-bundle\") pod \"ab9338b9-3a97-4139-aadb-6b22cf149616\" (UID: \"ab9338b9-3a97-4139-aadb-6b22cf149616\") "
Sep 17 19:13:55.602742 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:55.602731    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9338b9-3a97-4139-aadb-6b22cf149616-proxy-tls\") pod \"ab9338b9-3a97-4139-aadb-6b22cf149616\" (UID: \"ab9338b9-3a97-4139-aadb-6b22cf149616\") "
Sep 17 19:13:55.602974 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:55.602955    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9338b9-3a97-4139-aadb-6b22cf149616-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ab9338b9-3a97-4139-aadb-6b22cf149616" (UID: "ab9338b9-3a97-4139-aadb-6b22cf149616"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:55.604715 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:55.604696    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab9338b9-3a97-4139-aadb-6b22cf149616-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab9338b9-3a97-4139-aadb-6b22cf149616" (UID: "ab9338b9-3a97-4139-aadb-6b22cf149616"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:55.703946 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:55.703911    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab9338b9-3a97-4139-aadb-6b22cf149616-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:13:55.703946 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:55.703942    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab9338b9-3a97-4139-aadb-6b22cf149616-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:13:56.058816 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.058777    2557 generic.go:356] "Generic (PLEG): container finished" podID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerID="d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1" exitCode=0
Sep 17 19:13:56.059016 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.058828    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"
Sep 17 19:13:56.059016 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.058833    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" event={"ID":"ab9338b9-3a97-4139-aadb-6b22cf149616","Type":"ContainerDied","Data":"d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1"}
Sep 17 19:13:56.059016 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.058948    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc" event={"ID":"ab9338b9-3a97-4139-aadb-6b22cf149616","Type":"ContainerDied","Data":"5455bd86a3a63edeb848018b4bf35a424e313b2bceeb2aff5ccb37909bbbc71b"}
Sep 17 19:13:56.059016 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.058972    2557 scope.go:117] "RemoveContainer" containerID="d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1"
Sep 17 19:13:56.067773 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.067752    2557 scope.go:117] "RemoveContainer" containerID="d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1"
Sep 17 19:13:56.068062 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:13:56.068041    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1\": container with ID starting with d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1 not found: ID does not exist" containerID="d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1"
Sep 17 19:13:56.068121 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.068076    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1"} err="failed to get container status \"d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1\": rpc error: code = NotFound desc = could not find container \"d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1\": container with ID starting with d0671e29ba4a3acb854c09eb4eee2da5ac881e9882e48b35ec11f2a1f19796d1 not found: ID does not exist"
Sep 17 19:13:56.080870 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.080844    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"]
Sep 17 19:13:56.084392 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.084370    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bd292-8446b7b59d-5t2pc"]
Sep 17 19:13:56.835141 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:13:56.835110    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" path="/var/lib/kubelet/pods/ab9338b9-3a97-4139-aadb-6b22cf149616/volumes"
Sep 17 19:14:01.843433 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:01.843401    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"]
Sep 17 19:14:01.843816 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:01.843601    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2" containerID="cri-o://6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781" gracePeriod=30
Sep 17 19:14:01.960981 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:01.960949    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"]
Sep 17 19:14:01.961190 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:01.961162    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container" containerID="cri-o://abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3" gracePeriod=30
Sep 17 19:14:01.961262 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:01.961199    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kube-rbac-proxy" containerID="cri-o://6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c" gracePeriod=30
Sep 17 19:14:02.092655 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.092624    2557 generic.go:356] "Generic (PLEG): container finished" podID="569ab150-dba1-460c-b101-b7044e1cfb47" containerID="6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c" exitCode=2
Sep 17 19:14:02.092809 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.092704    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" event={"ID":"569ab150-dba1-460c-b101-b7044e1cfb47","Type":"ContainerDied","Data":"6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c"}
Sep 17 19:14:02.120241 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120155    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"]
Sep 17 19:14:02.120512 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120486    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kube-rbac-proxy"
Sep 17 19:14:02.120512 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120498    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kube-rbac-proxy"
Sep 17 19:14:02.120512 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120509    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container"
Sep 17 19:14:02.120512 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120514    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container"
Sep 17 19:14:02.120652 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120530    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292"
Sep 17 19:14:02.120652 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120535    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292"
Sep 17 19:14:02.120652 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120585    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kube-rbac-proxy"
Sep 17 19:14:02.120652 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120593    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab9338b9-3a97-4139-aadb-6b22cf149616" containerName="ensemble-graph-bd292"
Sep 17 19:14:02.120652 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.120600    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="269ba13f-3f67-453c-a0f2-5300a3703170" containerName="kserve-container"
Sep 17 19:14:02.123648 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.123627    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.126652 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.126627    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e2192-predictor-serving-cert\""
Sep 17 19:14:02.126752 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.126673    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e2192-kube-rbac-proxy-sar-config\""
Sep 17 19:14:02.141675 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.141649    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"]
Sep 17 19:14:02.148288 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.148263    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc99v\" (UniqueName: \"kubernetes.io/projected/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-api-access-nc99v\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.148412 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.148332    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.148412 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.148366    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b09297a-af93-4b3a-9870-ef5603ce94fd-proxy-tls\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.249207 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.249167    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.249420 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.249240    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b09297a-af93-4b3a-9870-ef5603ce94fd-proxy-tls\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.249420 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.249283    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc99v\" (UniqueName: \"kubernetes.io/projected/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-api-access-nc99v\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.249545 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:14:02.249421    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e2192-predictor-serving-cert: secret "success-200-isvc-e2192-predictor-serving-cert" not found
Sep 17 19:14:02.249545 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:14:02.249496    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b09297a-af93-4b3a-9870-ef5603ce94fd-proxy-tls podName:6b09297a-af93-4b3a-9870-ef5603ce94fd nodeName:}" failed. No retries permitted until 2026-09-17 19:14:02.749473237 +0000 UTC m=+2116.104732003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6b09297a-af93-4b3a-9870-ef5603ce94fd-proxy-tls") pod "success-200-isvc-e2192-predictor-79886769c8-v9dc9" (UID: "6b09297a-af93-4b3a-9870-ef5603ce94fd") : secret "success-200-isvc-e2192-predictor-serving-cert" not found
Sep 17 19:14:02.249864 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.249844    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.261326 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.261297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc99v\" (UniqueName: \"kubernetes.io/projected/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-api-access-nc99v\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.753403 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.753354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b09297a-af93-4b3a-9870-ef5603ce94fd-proxy-tls\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.755765 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.755736    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b09297a-af93-4b3a-9870-ef5603ce94fd-proxy-tls\") pod \"success-200-isvc-e2192-predictor-79886769c8-v9dc9\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") " pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:02.947339 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:02.947300    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 19:14:03.033628 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:03.033538    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:03.305389 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:03.305361    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"]
Sep 17 19:14:03.307876 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:14:03.307846    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b09297a_af93_4b3a_9870_ef5603ce94fd.slice/crio-747a994254dba174e64d05e7ed12aa9f167635c88f041aa6d81e14fd6da50fe8 WatchSource:0}: Error finding container 747a994254dba174e64d05e7ed12aa9f167635c88f041aa6d81e14fd6da50fe8: Status 404 returned error can't find the container with id 747a994254dba174e64d05e7ed12aa9f167635c88f041aa6d81e14fd6da50fe8
Sep 17 19:14:04.104830 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:04.104794    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" event={"ID":"6b09297a-af93-4b3a-9870-ef5603ce94fd","Type":"ContainerStarted","Data":"1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171"}
Sep 17 19:14:04.105246 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:04.104856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" event={"ID":"6b09297a-af93-4b3a-9870-ef5603ce94fd","Type":"ContainerStarted","Data":"d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c"}
Sep 17 19:14:04.105246 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:04.104882    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" event={"ID":"6b09297a-af93-4b3a-9870-ef5603ce94fd","Type":"ContainerStarted","Data":"747a994254dba174e64d05e7ed12aa9f167635c88f041aa6d81e14fd6da50fe8"}
Sep 17 19:14:04.105246 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:04.104950    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:04.125978 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:04.125908    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" podStartSLOduration=2.125886322 podStartE2EDuration="2.125886322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:04.123977485 +0000 UTC m=+2117.479236272" watchObservedRunningTime="2026-09-17 19:14:04.125886322 +0000 UTC m=+2117.481145098"
Sep 17 19:14:04.136816 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:04.136780    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:04.881130 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:04.881088    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 19:14:04.886582 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:04.886556    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 19:14:05.109412 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.109379    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:05.110751 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.110724    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:14:05.310265 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.310237    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:14:05.375419 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.375379    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/569ab150-dba1-460c-b101-b7044e1cfb47-kube-rbac-proxy-sar-config\") pod \"569ab150-dba1-460c-b101-b7044e1cfb47\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") "
Sep 17 19:14:05.375419 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.375427    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl2fz\" (UniqueName: \"kubernetes.io/projected/569ab150-dba1-460c-b101-b7044e1cfb47-kube-api-access-nl2fz\") pod \"569ab150-dba1-460c-b101-b7044e1cfb47\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") "
Sep 17 19:14:05.375596 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.375465    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/569ab150-dba1-460c-b101-b7044e1cfb47-proxy-tls\") pod \"569ab150-dba1-460c-b101-b7044e1cfb47\" (UID: \"569ab150-dba1-460c-b101-b7044e1cfb47\") "
Sep 17 19:14:05.375861 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.375833    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569ab150-dba1-460c-b101-b7044e1cfb47-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "569ab150-dba1-460c-b101-b7044e1cfb47" (UID: "569ab150-dba1-460c-b101-b7044e1cfb47"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:14:05.377662 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.377639    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569ab150-dba1-460c-b101-b7044e1cfb47-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "569ab150-dba1-460c-b101-b7044e1cfb47" (UID: "569ab150-dba1-460c-b101-b7044e1cfb47"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:14:05.377772 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.377697    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569ab150-dba1-460c-b101-b7044e1cfb47-kube-api-access-nl2fz" (OuterVolumeSpecName: "kube-api-access-nl2fz") pod "569ab150-dba1-460c-b101-b7044e1cfb47" (UID: "569ab150-dba1-460c-b101-b7044e1cfb47"). InnerVolumeSpecName "kube-api-access-nl2fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:05.476991 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.476886    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2fz\" (UniqueName: \"kubernetes.io/projected/569ab150-dba1-460c-b101-b7044e1cfb47-kube-api-access-nl2fz\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:14:05.476991 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.476935    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/569ab150-dba1-460c-b101-b7044e1cfb47-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:14:05.476991 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:05.476945    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/569ab150-dba1-460c-b101-b7044e1cfb47-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:14:06.114289 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.114257    2557 generic.go:356] "Generic (PLEG): container finished" podID="569ab150-dba1-460c-b101-b7044e1cfb47" containerID="abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3" exitCode=0
Sep 17 19:14:06.114723 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.114306    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"
Sep 17 19:14:06.114723 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.114341    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" event={"ID":"569ab150-dba1-460c-b101-b7044e1cfb47","Type":"ContainerDied","Data":"abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3"}
Sep 17 19:14:06.114723 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.114410    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk" event={"ID":"569ab150-dba1-460c-b101-b7044e1cfb47","Type":"ContainerDied","Data":"a7c5c5818db0360f955acbff0dc672bbcf5ad53c4c4aead91571db4eba125190"}
Sep 17 19:14:06.114723 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.114448    2557 scope.go:117] "RemoveContainer" containerID="6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c"
Sep 17 19:14:06.114943 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.114888    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:14:06.122951 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.122931    2557 scope.go:117] "RemoveContainer" containerID="abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3"
Sep 17 19:14:06.131611 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.131588    2557 scope.go:117] "RemoveContainer" containerID="6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c"
Sep 17 19:14:06.131957 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:14:06.131937    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c\": container with ID starting with 6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c not found: ID does not exist" containerID="6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c"
Sep 17 19:14:06.131999 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.131967    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c"} err="failed to get container status \"6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c\": rpc error: code = NotFound desc = could not find container \"6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c\": container with ID starting with 6d02ebfe6182e7dba88c81586e973cd6e78871d60da56487d5929cefacfb286c not found: ID does not exist"
Sep 17 19:14:06.131999 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.131985    2557 scope.go:117] "RemoveContainer" containerID="abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3"
Sep 17 19:14:06.132291 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:14:06.132273    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3\": container with ID starting with abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3 not found: ID does not exist" containerID="abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3"
Sep 17 19:14:06.132358 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.132295    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3"} err="failed to get container status \"abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3\": rpc error: code = NotFound desc = could not find container \"abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3\": container with ID starting with abb220f013bbccc146b364ae5d93393bdf8dc1dc1df4f8b963a28a2d8afb43b3 not found: ID does not exist"
Sep 17 19:14:06.135986 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.135960    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"]
Sep 17 19:14:06.139250 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.139225    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-058d2-predictor-6df78f8b6-drsmk"]
Sep 17 19:14:06.835632 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:06.835601    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" path="/var/lib/kubelet/pods/569ab150-dba1-460c-b101-b7044e1cfb47/volumes"
Sep 17 19:14:09.136730 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:09.136695    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:11.121734 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:11.121707    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:11.122206 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:11.122176    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:14:12.948194 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:12.948166    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:14:14.137651 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:14.137616    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:14.138054 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:14.137711    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:14:19.137592 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:19.137540    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:21.123120 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:21.123082    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:14:24.137001 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:24.136962    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:25.712914 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.708650    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"]
Sep 17 19:14:25.712914 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.709408    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container"
Sep 17 19:14:25.712914 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.709428    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container"
Sep 17 19:14:25.712914 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.709461    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kube-rbac-proxy"
Sep 17 19:14:25.712914 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.709478    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kube-rbac-proxy"
Sep 17 19:14:25.712914 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.709639    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kserve-container"
Sep 17 19:14:25.712914 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.709653    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="569ab150-dba1-460c-b101-b7044e1cfb47" containerName="kube-rbac-proxy"
Sep 17 19:14:25.713718 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.713413    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:25.716286 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.716261    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-3efeb-kube-rbac-proxy-sar-config\""
Sep 17 19:14:25.716405 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.716289    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-3efeb-serving-cert\""
Sep 17 19:14:25.719375 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.719352    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"]
Sep 17 19:14:25.835422 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.835392    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054d9702-58f7-4d25-9044-dbad9db30be8-openshift-service-ca-bundle\") pod \"splitter-graph-3efeb-c7897dd95-9nm5f\" (UID: \"054d9702-58f7-4d25-9044-dbad9db30be8\") " pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:25.835572 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.835442    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054d9702-58f7-4d25-9044-dbad9db30be8-proxy-tls\") pod \"splitter-graph-3efeb-c7897dd95-9nm5f\" (UID: \"054d9702-58f7-4d25-9044-dbad9db30be8\") " pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:25.936275 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.936235    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054d9702-58f7-4d25-9044-dbad9db30be8-proxy-tls\") pod \"splitter-graph-3efeb-c7897dd95-9nm5f\" (UID: \"054d9702-58f7-4d25-9044-dbad9db30be8\") " pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:25.936462 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:14:25.936330    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-3efeb-serving-cert: secret "splitter-graph-3efeb-serving-cert" not found
Sep 17 19:14:25.936462 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.936344    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054d9702-58f7-4d25-9044-dbad9db30be8-openshift-service-ca-bundle\") pod \"splitter-graph-3efeb-c7897dd95-9nm5f\" (UID: \"054d9702-58f7-4d25-9044-dbad9db30be8\") " pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:25.936462 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:14:25.936403    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/054d9702-58f7-4d25-9044-dbad9db30be8-proxy-tls podName:054d9702-58f7-4d25-9044-dbad9db30be8 nodeName:}" failed. No retries permitted until 2026-09-17 19:14:26.436382998 +0000 UTC m=+2139.791641751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/054d9702-58f7-4d25-9044-dbad9db30be8-proxy-tls") pod "splitter-graph-3efeb-c7897dd95-9nm5f" (UID: "054d9702-58f7-4d25-9044-dbad9db30be8") : secret "splitter-graph-3efeb-serving-cert" not found
Sep 17 19:14:25.936995 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:25.936979    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054d9702-58f7-4d25-9044-dbad9db30be8-openshift-service-ca-bundle\") pod \"splitter-graph-3efeb-c7897dd95-9nm5f\" (UID: \"054d9702-58f7-4d25-9044-dbad9db30be8\") " pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:26.440849 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:26.440812    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054d9702-58f7-4d25-9044-dbad9db30be8-proxy-tls\") pod \"splitter-graph-3efeb-c7897dd95-9nm5f\" (UID: \"054d9702-58f7-4d25-9044-dbad9db30be8\") " pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:26.443302 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:26.443270    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054d9702-58f7-4d25-9044-dbad9db30be8-proxy-tls\") pod \"splitter-graph-3efeb-c7897dd95-9nm5f\" (UID: \"054d9702-58f7-4d25-9044-dbad9db30be8\") " pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:26.625139 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:26.625106    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:26.896853 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:26.896830    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"]
Sep 17 19:14:26.899063 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:14:26.899038    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054d9702_58f7_4d25_9044_dbad9db30be8.slice/crio-de57e75791e9b48218c30a4d3724dad0b7fa2b0ebfc1d1e021918880771c028a WatchSource:0}: Error finding container de57e75791e9b48218c30a4d3724dad0b7fa2b0ebfc1d1e021918880771c028a: Status 404 returned error can't find the container with id de57e75791e9b48218c30a4d3724dad0b7fa2b0ebfc1d1e021918880771c028a
Sep 17 19:14:27.201886 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:27.201791    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" event={"ID":"054d9702-58f7-4d25-9044-dbad9db30be8","Type":"ContainerStarted","Data":"469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef"}
Sep 17 19:14:27.201886 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:27.201840    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" event={"ID":"054d9702-58f7-4d25-9044-dbad9db30be8","Type":"ContainerStarted","Data":"de57e75791e9b48218c30a4d3724dad0b7fa2b0ebfc1d1e021918880771c028a"}
Sep 17 19:14:27.202078 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:27.201897    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:27.220219 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:27.220170    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" podStartSLOduration=2.220155934 podStartE2EDuration="2.220155934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:27.218724823 +0000 UTC m=+2140.573983608" watchObservedRunningTime="2026-09-17 19:14:27.220155934 +0000 UTC m=+2140.575414707"
Sep 17 19:14:29.137316 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:29.137280    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:31.122326 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:31.122245    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:14:31.989899 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:31.989870    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:14:32.082298 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.082266    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3580520d-06dd-432e-8a6c-1f5f4d2f5996-openshift-service-ca-bundle\") pod \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\" (UID: \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\") "
Sep 17 19:14:32.082479 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.082358    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3580520d-06dd-432e-8a6c-1f5f4d2f5996-proxy-tls\") pod \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\" (UID: \"3580520d-06dd-432e-8a6c-1f5f4d2f5996\") "
Sep 17 19:14:32.082685 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.082666    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3580520d-06dd-432e-8a6c-1f5f4d2f5996-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3580520d-06dd-432e-8a6c-1f5f4d2f5996" (UID: "3580520d-06dd-432e-8a6c-1f5f4d2f5996"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:14:32.084448 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.084424    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3580520d-06dd-432e-8a6c-1f5f4d2f5996-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3580520d-06dd-432e-8a6c-1f5f4d2f5996" (UID: "3580520d-06dd-432e-8a6c-1f5f4d2f5996"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:14:32.183537 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.183454    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3580520d-06dd-432e-8a6c-1f5f4d2f5996-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:14:32.183537 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.183480    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3580520d-06dd-432e-8a6c-1f5f4d2f5996-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:14:32.224837 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.224800    2557 generic.go:356] "Generic (PLEG): container finished" podID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerID="6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781" exitCode=0
Sep 17 19:14:32.225017 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.224845    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" event={"ID":"3580520d-06dd-432e-8a6c-1f5f4d2f5996","Type":"ContainerDied","Data":"6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781"}
Sep 17 19:14:32.225017 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.224858    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"
Sep 17 19:14:32.225017 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.224928    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh" event={"ID":"3580520d-06dd-432e-8a6c-1f5f4d2f5996","Type":"ContainerDied","Data":"3c81150e94d6f5921dd0665b7c9ada3aac8104f0aaa2d96679968700cf652315"}
Sep 17 19:14:32.225017 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.224961    2557 scope.go:117] "RemoveContainer" containerID="6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781"
Sep 17 19:14:32.236366 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.236339    2557 scope.go:117] "RemoveContainer" containerID="6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781"
Sep 17 19:14:32.236866 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:14:32.236840    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781\": container with ID starting with 6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781 not found: ID does not exist" containerID="6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781"
Sep 17 19:14:32.236959 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.236880    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781"} err="failed to get container status \"6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781\": rpc error: code = NotFound desc = could not find container \"6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781\": container with ID starting with 6ef38bdabc4bfbe52811d31bfbbd7d815c28dfacb819c8f0af0f42c086687781 not found: ID does not exist"
Sep 17 19:14:32.247767 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.247744    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"]
Sep 17 19:14:32.253888 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.253864    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-058d2-ff64f56f7-mmrgh"]
Sep 17 19:14:32.837245 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:32.837196    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" path="/var/lib/kubelet/pods/3580520d-06dd-432e-8a6c-1f5f4d2f5996/volumes"
Sep 17 19:14:33.212465 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:33.212371    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:35.787015 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:35.786928    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"]
Sep 17 19:14:35.787470 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:35.787179    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb" containerID="cri-o://469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef" gracePeriod=30
Sep 17 19:14:35.886870 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:35.886840    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"]
Sep 17 19:14:35.887092 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:35.887067    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container" containerID="cri-o://bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31" gracePeriod=30
Sep 17 19:14:35.887169 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:35.887093    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kube-rbac-proxy" containerID="cri-o://c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971" gracePeriod=30
Sep 17 19:14:36.076547 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.076514    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"]
Sep 17 19:14:36.076835 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.076821    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2"
Sep 17 19:14:36.076886 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.076838    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2"
Sep 17 19:14:36.076921 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.076914    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3580520d-06dd-432e-8a6c-1f5f4d2f5996" containerName="sequence-graph-058d2"
Sep 17 19:14:36.079974 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.079950    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.082693 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.082674    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-46c43-predictor-serving-cert\""
Sep 17 19:14:36.082824 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.082680    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-46c43-kube-rbac-proxy-sar-config\""
Sep 17 19:14:36.089666 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.089644    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"]
Sep 17 19:14:36.215759 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.215724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-46c43-predictor-569ff545b8-d5d4c\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") " pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.215759 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.215763    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcr88\" (UniqueName: \"kubernetes.io/projected/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-api-access-gcr88\") pod \"success-200-isvc-46c43-predictor-569ff545b8-d5d4c\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") " pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.215981 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.215807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8e63db-202a-448e-95ad-e27b3cd6e310-proxy-tls\") pod \"success-200-isvc-46c43-predictor-569ff545b8-d5d4c\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") " pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.242803 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.242767    2557 generic.go:356] "Generic (PLEG): container finished" podID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerID="c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971" exitCode=2
Sep 17 19:14:36.242956 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.242817    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" event={"ID":"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e","Type":"ContainerDied","Data":"c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971"}
Sep 17 19:14:36.316398 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.316360    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8e63db-202a-448e-95ad-e27b3cd6e310-proxy-tls\") pod \"success-200-isvc-46c43-predictor-569ff545b8-d5d4c\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") " pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.316613 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.316485    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-46c43-predictor-569ff545b8-d5d4c\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") " pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.316613 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.316524    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcr88\" (UniqueName: \"kubernetes.io/projected/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-api-access-gcr88\") pod \"success-200-isvc-46c43-predictor-569ff545b8-d5d4c\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") " pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.317340 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.317308    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-46c43-predictor-569ff545b8-d5d4c\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") " pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.318998 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.318980    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8e63db-202a-448e-95ad-e27b3cd6e310-proxy-tls\") pod \"success-200-isvc-46c43-predictor-569ff545b8-d5d4c\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") " pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.324712 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.324684    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcr88\" (UniqueName: \"kubernetes.io/projected/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-api-access-gcr88\") pod \"success-200-isvc-46c43-predictor-569ff545b8-d5d4c\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") " pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.391495 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.391412    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:36.664440 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:36.664414    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"]
Sep 17 19:14:36.666346 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:14:36.666321    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8e63db_202a_448e_95ad_e27b3cd6e310.slice/crio-cda1a24edc3fb7c26178cd3873f6ef6f07eb6eaa33a4be226db3e514bb70e937 WatchSource:0}: Error finding container cda1a24edc3fb7c26178cd3873f6ef6f07eb6eaa33a4be226db3e514bb70e937: Status 404 returned error can't find the container with id cda1a24edc3fb7c26178cd3873f6ef6f07eb6eaa33a4be226db3e514bb70e937
Sep 17 19:14:37.249133 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:37.249091    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" event={"ID":"2c8e63db-202a-448e-95ad-e27b3cd6e310","Type":"ContainerStarted","Data":"5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e"}
Sep 17 19:14:37.249598 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:37.249167    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" event={"ID":"2c8e63db-202a-448e-95ad-e27b3cd6e310","Type":"ContainerStarted","Data":"cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f"}
Sep 17 19:14:37.249598 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:37.249201    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" event={"ID":"2c8e63db-202a-448e-95ad-e27b3cd6e310","Type":"ContainerStarted","Data":"cda1a24edc3fb7c26178cd3873f6ef6f07eb6eaa33a4be226db3e514bb70e937"}
Sep 17 19:14:37.249598 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:37.249251    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:37.249598 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:37.249284    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:37.250576 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:37.250551    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 19:14:37.269775 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:37.269731    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podStartSLOduration=1.269717693 podStartE2EDuration="1.269717693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:37.267493894 +0000 UTC m=+2150.622752689" watchObservedRunningTime="2026-09-17 19:14:37.269717693 +0000 UTC m=+2150.624976466"
Sep 17 19:14:37.942355 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:37.942303    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 17 19:14:38.210878 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:38.210791    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:38.253776 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:38.253738    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 19:14:39.242748 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.242726    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:14:39.258994 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.258961    2557 generic.go:356] "Generic (PLEG): container finished" podID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerID="bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31" exitCode=0
Sep 17 19:14:39.259393 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.259011    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"
Sep 17 19:14:39.259393 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.259052    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" event={"ID":"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e","Type":"ContainerDied","Data":"bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31"}
Sep 17 19:14:39.259393 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.259128    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5" event={"ID":"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e","Type":"ContainerDied","Data":"f00e26d2f5cc07667845676c99bf2953ace5db13f849f3806cab7b90cbbd3a33"}
Sep 17 19:14:39.259393 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.259167    2557 scope.go:117] "RemoveContainer" containerID="c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971"
Sep 17 19:14:39.268148 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.268130    2557 scope.go:117] "RemoveContainer" containerID="bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31"
Sep 17 19:14:39.276332 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.276304    2557 scope.go:117] "RemoveContainer" containerID="c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971"
Sep 17 19:14:39.276632 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:14:39.276612    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971\": container with ID starting with c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971 not found: ID does not exist" containerID="c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971"
Sep 17 19:14:39.276685 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.276641    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971"} err="failed to get container status \"c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971\": rpc error: code = NotFound desc = could not find container \"c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971\": container with ID starting with c8f0517deab9abd4a29dd2cdd34ebfa870fecade842b1ba79119938edd1f9971 not found: ID does not exist"
Sep 17 19:14:39.276685 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.276658    2557 scope.go:117] "RemoveContainer" containerID="bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31"
Sep 17 19:14:39.276936 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:14:39.276921    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31\": container with ID starting with bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31 not found: ID does not exist" containerID="bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31"
Sep 17 19:14:39.277006 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.276943    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31"} err="failed to get container status \"bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31\": rpc error: code = NotFound desc = could not find container \"bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31\": container with ID starting with bae02a545091ec7c996b93aa6ddbac11b2eb241252dd9c4c7a5ffae6679dee31 not found: ID does not exist"
Sep 17 19:14:39.343705 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.343672    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-proxy-tls\") pod \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") "
Sep 17 19:14:39.343705 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.343709    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-rbac-proxy-sar-config\") pod \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") "
Sep 17 19:14:39.343923 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.343739    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px788\" (UniqueName: \"kubernetes.io/projected/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-api-access-px788\") pod \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\" (UID: \"60c8ed24-e4e7-4809-a76a-6c49f54bbb1e\") "
Sep 17 19:14:39.344123 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.344100    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" (UID: "60c8ed24-e4e7-4809-a76a-6c49f54bbb1e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:14:39.345918 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.345892    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" (UID: "60c8ed24-e4e7-4809-a76a-6c49f54bbb1e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:14:39.346040 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.345896    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-api-access-px788" (OuterVolumeSpecName: "kube-api-access-px788") pod "60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" (UID: "60c8ed24-e4e7-4809-a76a-6c49f54bbb1e"). InnerVolumeSpecName "kube-api-access-px788". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:39.445331 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.445295    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:14:39.445331 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.445325    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:14:39.445331 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.445336    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px788\" (UniqueName: \"kubernetes.io/projected/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e-kube-api-access-px788\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:14:39.580774 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.580739    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"]
Sep 17 19:14:39.585743 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:39.585716    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3efeb-predictor-9b86cfccd-4z9t5"]
Sep 17 19:14:40.835074 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:40.835040    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" path="/var/lib/kubelet/pods/60c8ed24-e4e7-4809-a76a-6c49f54bbb1e/volumes"
Sep 17 19:14:41.123224 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:41.123110    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 19:14:43.211009 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:43.210966    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:43.259844 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:43.259817    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:14:43.260316 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:43.260289    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 19:14:48.211444 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:48.211399    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:48.211903 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:48.211500    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:14:51.123411 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:51.123371    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:14:53.210611 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:53.210565    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:14:53.260458 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:53.260410    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 19:14:58.211389 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:14:58.211346    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:15:02.077683 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.077650    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"]
Sep 17 19:15:02.078066 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.077988    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kube-rbac-proxy"
Sep 17 19:15:02.078066 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.078003    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kube-rbac-proxy"
Sep 17 19:15:02.078066 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.078015    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container"
Sep 17 19:15:02.078066 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.078021    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container"
Sep 17 19:15:02.078195 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.078079    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kube-rbac-proxy"
Sep 17 19:15:02.078195 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.078089    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c8ed24-e4e7-4809-a76a-6c49f54bbb1e" containerName="kserve-container"
Sep 17 19:15:02.082468 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.082451    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:02.085164 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.085134    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-e2192-kube-rbac-proxy-sar-config\""
Sep 17 19:15:02.085312 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.085254    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-e2192-serving-cert\""
Sep 17 19:15:02.091305 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.091279    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"]
Sep 17 19:15:02.230636 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.230592    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07c1ede4-84f8-48db-9888-2b93d021dc55-openshift-service-ca-bundle\") pod \"switch-graph-e2192-7f99675849-4dd6h\" (UID: \"07c1ede4-84f8-48db-9888-2b93d021dc55\") " pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:02.230636 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.230639    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c1ede4-84f8-48db-9888-2b93d021dc55-proxy-tls\") pod \"switch-graph-e2192-7f99675849-4dd6h\" (UID: \"07c1ede4-84f8-48db-9888-2b93d021dc55\") " pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:02.331474 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.331385    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07c1ede4-84f8-48db-9888-2b93d021dc55-openshift-service-ca-bundle\") pod \"switch-graph-e2192-7f99675849-4dd6h\" (UID: \"07c1ede4-84f8-48db-9888-2b93d021dc55\") " pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:02.331474 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.331424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c1ede4-84f8-48db-9888-2b93d021dc55-proxy-tls\") pod \"switch-graph-e2192-7f99675849-4dd6h\" (UID: \"07c1ede4-84f8-48db-9888-2b93d021dc55\") " pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:02.332341 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.332313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07c1ede4-84f8-48db-9888-2b93d021dc55-openshift-service-ca-bundle\") pod \"switch-graph-e2192-7f99675849-4dd6h\" (UID: \"07c1ede4-84f8-48db-9888-2b93d021dc55\") " pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:02.333929 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.333907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c1ede4-84f8-48db-9888-2b93d021dc55-proxy-tls\") pod \"switch-graph-e2192-7f99675849-4dd6h\" (UID: \"07c1ede4-84f8-48db-9888-2b93d021dc55\") " pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:02.393436 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.393405    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:02.664658 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:02.664634    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"]
Sep 17 19:15:02.667274 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:15:02.667245    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c1ede4_84f8_48db_9888_2b93d021dc55.slice/crio-2fe03fbee97c5a250a5d8dc9a7855abb1689912f58b3d7c182c5db3056962c6c WatchSource:0}: Error finding container 2fe03fbee97c5a250a5d8dc9a7855abb1689912f58b3d7c182c5db3056962c6c: Status 404 returned error can't find the container with id 2fe03fbee97c5a250a5d8dc9a7855abb1689912f58b3d7c182c5db3056962c6c
Sep 17 19:15:03.210702 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:03.210662    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:15:03.260339 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:03.260296    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 19:15:03.359202 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:03.359161    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" event={"ID":"07c1ede4-84f8-48db-9888-2b93d021dc55","Type":"ContainerStarted","Data":"5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d"}
Sep 17 19:15:03.359398 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:03.359244    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" event={"ID":"07c1ede4-84f8-48db-9888-2b93d021dc55","Type":"ContainerStarted","Data":"2fe03fbee97c5a250a5d8dc9a7855abb1689912f58b3d7c182c5db3056962c6c"}
Sep 17 19:15:03.359398 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:03.359329    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:03.379195 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:03.379144    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" podStartSLOduration=1.379128253 podStartE2EDuration="1.379128253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:03.378324695 +0000 UTC m=+2176.733583501" watchObservedRunningTime="2026-09-17 19:15:03.379128253 +0000 UTC m=+2176.734387025"
Sep 17 19:15:05.934569 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:05.934540    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:15:05.960516 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:05.960489    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054d9702-58f7-4d25-9044-dbad9db30be8-proxy-tls\") pod \"054d9702-58f7-4d25-9044-dbad9db30be8\" (UID: \"054d9702-58f7-4d25-9044-dbad9db30be8\") "
Sep 17 19:15:05.960670 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:05.960524    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054d9702-58f7-4d25-9044-dbad9db30be8-openshift-service-ca-bundle\") pod \"054d9702-58f7-4d25-9044-dbad9db30be8\" (UID: \"054d9702-58f7-4d25-9044-dbad9db30be8\") "
Sep 17 19:15:05.960896 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:05.960876    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054d9702-58f7-4d25-9044-dbad9db30be8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "054d9702-58f7-4d25-9044-dbad9db30be8" (UID: "054d9702-58f7-4d25-9044-dbad9db30be8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:15:05.962583 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:05.962562    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054d9702-58f7-4d25-9044-dbad9db30be8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "054d9702-58f7-4d25-9044-dbad9db30be8" (UID: "054d9702-58f7-4d25-9044-dbad9db30be8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:15:06.061880 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.061783    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/054d9702-58f7-4d25-9044-dbad9db30be8-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:15:06.061880 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.061817    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054d9702-58f7-4d25-9044-dbad9db30be8-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:15:06.372019 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.371976    2557 generic.go:356] "Generic (PLEG): container finished" podID="054d9702-58f7-4d25-9044-dbad9db30be8" containerID="469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef" exitCode=0
Sep 17 19:15:06.372019 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.372026    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"
Sep 17 19:15:06.372270 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.372068    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" event={"ID":"054d9702-58f7-4d25-9044-dbad9db30be8","Type":"ContainerDied","Data":"469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef"}
Sep 17 19:15:06.372270 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.372129    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f" event={"ID":"054d9702-58f7-4d25-9044-dbad9db30be8","Type":"ContainerDied","Data":"de57e75791e9b48218c30a4d3724dad0b7fa2b0ebfc1d1e021918880771c028a"}
Sep 17 19:15:06.372270 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.372152    2557 scope.go:117] "RemoveContainer" containerID="469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef"
Sep 17 19:15:06.380616 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.380596    2557 scope.go:117] "RemoveContainer" containerID="469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef"
Sep 17 19:15:06.380884 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:15:06.380866    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef\": container with ID starting with 469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef not found: ID does not exist" containerID="469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef"
Sep 17 19:15:06.380961 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.380892    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef"} err="failed to get container status \"469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef\": rpc error: code = NotFound desc = could not find container \"469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef\": container with ID starting with 469c4e6f2ee596544ef18d4e1c0b38d65ee1159acefaa1da09282b43466a32ef not found: ID does not exist"
Sep 17 19:15:06.394675 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.394626    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"]
Sep 17 19:15:06.396048 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.396025    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-3efeb-c7897dd95-9nm5f"]
Sep 17 19:15:06.836131 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:06.836092    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" path="/var/lib/kubelet/pods/054d9702-58f7-4d25-9044-dbad9db30be8/volumes"
Sep 17 19:15:09.369127 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:09.369090    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:15:13.260317 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:13.260273    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 19:15:23.261374 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:23.261346    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:15:36.028187 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.028155    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"]
Sep 17 19:15:36.028619 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.028481    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb"
Sep 17 19:15:36.028619 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.028493    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb"
Sep 17 19:15:36.028619 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.028570    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="054d9702-58f7-4d25-9044-dbad9db30be8" containerName="splitter-graph-3efeb"
Sep 17 19:15:36.031449 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.031434    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:36.034341 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.034317    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-46c43-serving-cert\""
Sep 17 19:15:36.034520 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.034505    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-46c43-kube-rbac-proxy-sar-config\""
Sep 17 19:15:36.040931 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.040911    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"]
Sep 17 19:15:36.093970 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.093931    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb16c91e-ff28-4218-9530-d93855483151-openshift-service-ca-bundle\") pod \"splitter-graph-46c43-774db8878c-6vckc\" (UID: \"eb16c91e-ff28-4218-9530-d93855483151\") " pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:36.093970 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.093982    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb16c91e-ff28-4218-9530-d93855483151-proxy-tls\") pod \"splitter-graph-46c43-774db8878c-6vckc\" (UID: \"eb16c91e-ff28-4218-9530-d93855483151\") " pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:36.194850 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.194812    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb16c91e-ff28-4218-9530-d93855483151-openshift-service-ca-bundle\") pod \"splitter-graph-46c43-774db8878c-6vckc\" (UID: \"eb16c91e-ff28-4218-9530-d93855483151\") " pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:36.195050 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.194859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb16c91e-ff28-4218-9530-d93855483151-proxy-tls\") pod \"splitter-graph-46c43-774db8878c-6vckc\" (UID: \"eb16c91e-ff28-4218-9530-d93855483151\") " pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:36.195050 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:15:36.195025    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-46c43-serving-cert: secret "splitter-graph-46c43-serving-cert" not found
Sep 17 19:15:36.195129 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:15:36.195088    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb16c91e-ff28-4218-9530-d93855483151-proxy-tls podName:eb16c91e-ff28-4218-9530-d93855483151 nodeName:}" failed. No retries permitted until 2026-09-17 19:15:36.69507187 +0000 UTC m=+2210.050330639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eb16c91e-ff28-4218-9530-d93855483151-proxy-tls") pod "splitter-graph-46c43-774db8878c-6vckc" (UID: "eb16c91e-ff28-4218-9530-d93855483151") : secret "splitter-graph-46c43-serving-cert" not found
Sep 17 19:15:36.195459 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.195441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb16c91e-ff28-4218-9530-d93855483151-openshift-service-ca-bundle\") pod \"splitter-graph-46c43-774db8878c-6vckc\" (UID: \"eb16c91e-ff28-4218-9530-d93855483151\") " pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:36.698197 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.698144    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb16c91e-ff28-4218-9530-d93855483151-proxy-tls\") pod \"splitter-graph-46c43-774db8878c-6vckc\" (UID: \"eb16c91e-ff28-4218-9530-d93855483151\") " pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:36.700717 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.700693    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb16c91e-ff28-4218-9530-d93855483151-proxy-tls\") pod \"splitter-graph-46c43-774db8878c-6vckc\" (UID: \"eb16c91e-ff28-4218-9530-d93855483151\") " pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:36.941130 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:36.941095    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:37.211571 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:37.211544    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"]
Sep 17 19:15:37.214206 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:15:37.214181    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb16c91e_ff28_4218_9530_d93855483151.slice/crio-d8448bb0a5695e821bc6d915e62b6f50b99aad7ab3b0fc3f254ae6aced2961b6 WatchSource:0}: Error finding container d8448bb0a5695e821bc6d915e62b6f50b99aad7ab3b0fc3f254ae6aced2961b6: Status 404 returned error can't find the container with id d8448bb0a5695e821bc6d915e62b6f50b99aad7ab3b0fc3f254ae6aced2961b6
Sep 17 19:15:37.499232 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:37.499130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" event={"ID":"eb16c91e-ff28-4218-9530-d93855483151","Type":"ContainerStarted","Data":"8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37"}
Sep 17 19:15:37.499232 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:37.499183    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" event={"ID":"eb16c91e-ff28-4218-9530-d93855483151","Type":"ContainerStarted","Data":"d8448bb0a5695e821bc6d915e62b6f50b99aad7ab3b0fc3f254ae6aced2961b6"}
Sep 17 19:15:37.499232 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:37.499229    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:15:37.518604 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:37.518554    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" podStartSLOduration=1.5185390170000002 podStartE2EDuration="1.518539017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:37.51593208 +0000 UTC m=+2210.871190854" watchObservedRunningTime="2026-09-17 19:15:37.518539017 +0000 UTC m=+2210.873797790"
Sep 17 19:15:43.508830 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:15:43.508801    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:18:48.868549 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:18:48.868522    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:18:48.871859 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:18:48.871841    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:18:48.872558 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:18:48.872537    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:18:48.875973 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:18:48.875955    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:23:48.893145 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:48.893030    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:23:48.897198 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:48.896055    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:23:48.897198 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:48.896960    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:23:48.900128 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:48.900111    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:23:50.739147 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:50.739114    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"]
Sep 17 19:23:50.739636 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:50.739387    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43" containerID="cri-o://8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37" gracePeriod=30
Sep 17 19:23:50.912147 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:50.912102    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"]
Sep 17 19:23:50.912471 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:50.912442    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container" containerID="cri-o://cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f" gracePeriod=30
Sep 17 19:23:50.912564 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:50.912489    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kube-rbac-proxy" containerID="cri-o://5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e" gracePeriod=30
Sep 17 19:23:51.421361 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:51.421316    2557 generic.go:356] "Generic (PLEG): container finished" podID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerID="5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e" exitCode=2
Sep 17 19:23:51.421534 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:51.421390    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" event={"ID":"2c8e63db-202a-448e-95ad-e27b3cd6e310","Type":"ContainerDied","Data":"5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e"}
Sep 17 19:23:53.254264 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:53.254194    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 19:23:53.260735 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:53.260707    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 19:23:53.507110 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:53.507022    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:23:54.166068 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.166046    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:23:54.266448 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.266359    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8e63db-202a-448e-95ad-e27b3cd6e310-proxy-tls\") pod \"2c8e63db-202a-448e-95ad-e27b3cd6e310\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") "
Sep 17 19:23:54.266448 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.266399    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-rbac-proxy-sar-config\") pod \"2c8e63db-202a-448e-95ad-e27b3cd6e310\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") "
Sep 17 19:23:54.266448 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.266435    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcr88\" (UniqueName: \"kubernetes.io/projected/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-api-access-gcr88\") pod \"2c8e63db-202a-448e-95ad-e27b3cd6e310\" (UID: \"2c8e63db-202a-448e-95ad-e27b3cd6e310\") "
Sep 17 19:23:54.266911 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.266806    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c8e63db-202a-448e-95ad-e27b3cd6e310" (UID: "2c8e63db-202a-448e-95ad-e27b3cd6e310"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:23:54.268523 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.268503    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8e63db-202a-448e-95ad-e27b3cd6e310-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c8e63db-202a-448e-95ad-e27b3cd6e310" (UID: "2c8e63db-202a-448e-95ad-e27b3cd6e310"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:54.268590 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.268499    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-api-access-gcr88" (OuterVolumeSpecName: "kube-api-access-gcr88") pod "2c8e63db-202a-448e-95ad-e27b3cd6e310" (UID: "2c8e63db-202a-448e-95ad-e27b3cd6e310"). InnerVolumeSpecName "kube-api-access-gcr88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:54.367617 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.367587    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8e63db-202a-448e-95ad-e27b3cd6e310-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:23:54.367617 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.367613    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:23:54.367617 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.367623    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcr88\" (UniqueName: \"kubernetes.io/projected/2c8e63db-202a-448e-95ad-e27b3cd6e310-kube-api-access-gcr88\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:23:54.435409 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.435376    2557 generic.go:356] "Generic (PLEG): container finished" podID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerID="cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f" exitCode=0
Sep 17 19:23:54.435600 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.435435    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"
Sep 17 19:23:54.435600 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.435467    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" event={"ID":"2c8e63db-202a-448e-95ad-e27b3cd6e310","Type":"ContainerDied","Data":"cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f"}
Sep 17 19:23:54.435600 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.435527    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c" event={"ID":"2c8e63db-202a-448e-95ad-e27b3cd6e310","Type":"ContainerDied","Data":"cda1a24edc3fb7c26178cd3873f6ef6f07eb6eaa33a4be226db3e514bb70e937"}
Sep 17 19:23:54.435600 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.435552    2557 scope.go:117] "RemoveContainer" containerID="5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e"
Sep 17 19:23:54.444368 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.444345    2557 scope.go:117] "RemoveContainer" containerID="cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f"
Sep 17 19:23:54.451920 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.451905    2557 scope.go:117] "RemoveContainer" containerID="5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e"
Sep 17 19:23:54.452174 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:23:54.452155    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e\": container with ID starting with 5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e not found: ID does not exist" containerID="5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e"
Sep 17 19:23:54.452263 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.452185    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e"} err="failed to get container status \"5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e\": rpc error: code = NotFound desc = could not find container \"5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e\": container with ID starting with 5e02b31817668a24b0ca8ee3780550a471c97df29bb5eee723cf532a7ae0ef5e not found: ID does not exist"
Sep 17 19:23:54.452370 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.452348    2557 scope.go:117] "RemoveContainer" containerID="cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f"
Sep 17 19:23:54.452626 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:23:54.452612    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f\": container with ID starting with cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f not found: ID does not exist" containerID="cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f"
Sep 17 19:23:54.452695 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.452630    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f"} err="failed to get container status \"cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f\": rpc error: code = NotFound desc = could not find container \"cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f\": container with ID starting with cda2d8f525a03d7df238d199ca2a18d258b532f0e6d1ff703915e125244d823f not found: ID does not exist"
Sep 17 19:23:54.463129 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.463101    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"]
Sep 17 19:23:54.466107 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.466070    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-46c43-predictor-569ff545b8-d5d4c"]
Sep 17 19:23:54.835624 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:54.835596    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" path="/var/lib/kubelet/pods/2c8e63db-202a-448e-95ad-e27b3cd6e310/volumes"
Sep 17 19:23:58.507875 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:23:58.507836    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:24:03.507502 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:03.507464    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:24:03.507911 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:03.507579    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:24:08.507889 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:08.507849    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:24:13.507561 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:13.507524    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:24:18.507167 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:18.507123    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:24:20.884639 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:20.884614    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:24:20.977364 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:20.977330    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb16c91e-ff28-4218-9530-d93855483151-proxy-tls\") pod \"eb16c91e-ff28-4218-9530-d93855483151\" (UID: \"eb16c91e-ff28-4218-9530-d93855483151\") "
Sep 17 19:24:20.977520 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:20.977380    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb16c91e-ff28-4218-9530-d93855483151-openshift-service-ca-bundle\") pod \"eb16c91e-ff28-4218-9530-d93855483151\" (UID: \"eb16c91e-ff28-4218-9530-d93855483151\") "
Sep 17 19:24:20.977778 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:20.977752    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb16c91e-ff28-4218-9530-d93855483151-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "eb16c91e-ff28-4218-9530-d93855483151" (UID: "eb16c91e-ff28-4218-9530-d93855483151"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:24:20.979337 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:20.979318    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb16c91e-ff28-4218-9530-d93855483151-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb16c91e-ff28-4218-9530-d93855483151" (UID: "eb16c91e-ff28-4218-9530-d93855483151"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:24:21.078876 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.078841    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb16c91e-ff28-4218-9530-d93855483151-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:24:21.078876 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.078868    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb16c91e-ff28-4218-9530-d93855483151-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:24:21.544011 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.543927    2557 generic.go:356] "Generic (PLEG): container finished" podID="eb16c91e-ff28-4218-9530-d93855483151" containerID="8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37" exitCode=0
Sep 17 19:24:21.544011 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.543971    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"
Sep 17 19:24:21.544193 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.544013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" event={"ID":"eb16c91e-ff28-4218-9530-d93855483151","Type":"ContainerDied","Data":"8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37"}
Sep 17 19:24:21.544193 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.544063    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc" event={"ID":"eb16c91e-ff28-4218-9530-d93855483151","Type":"ContainerDied","Data":"d8448bb0a5695e821bc6d915e62b6f50b99aad7ab3b0fc3f254ae6aced2961b6"}
Sep 17 19:24:21.544193 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.544087    2557 scope.go:117] "RemoveContainer" containerID="8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37"
Sep 17 19:24:21.552237 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.552201    2557 scope.go:117] "RemoveContainer" containerID="8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37"
Sep 17 19:24:21.552521 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:24:21.552503    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37\": container with ID starting with 8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37 not found: ID does not exist" containerID="8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37"
Sep 17 19:24:21.552574 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.552529    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37"} err="failed to get container status \"8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37\": rpc error: code = NotFound desc = could not find container \"8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37\": container with ID starting with 8334893b0db73b6040b0b219c93d33405573f49706059dbfc224c1dc083c4e37 not found: ID does not exist"
Sep 17 19:24:21.564981 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.564956    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"]
Sep 17 19:24:21.571060 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:21.571039    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-46c43-774db8878c-6vckc"]
Sep 17 19:24:22.835332 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:24:22.835299    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb16c91e-ff28-4218-9530-d93855483151" path="/var/lib/kubelet/pods/eb16c91e-ff28-4218-9530-d93855483151/volumes"
Sep 17 19:28:48.915444 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:28:48.915328    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:28:48.919372 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:28:48.918710    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:28:48.920915 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:28:48.920900    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:28:48.924052 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:28:48.924034    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:31:21.290052 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:21.289976    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"]
Sep 17 19:31:21.290557 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:21.290190    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192" containerID="cri-o://5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d" gracePeriod=30
Sep 17 19:31:21.427271 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:21.427235    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"]
Sep 17 19:31:21.427485 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:21.427456    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container" containerID="cri-o://d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c" gracePeriod=30
Sep 17 19:31:21.427573 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:21.427492    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kube-rbac-proxy" containerID="cri-o://1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171" gracePeriod=30
Sep 17 19:31:22.177137 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:22.177100    2557 generic.go:356] "Generic (PLEG): container finished" podID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerID="1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171" exitCode=2
Sep 17 19:31:22.177343 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:22.177168    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" event={"ID":"6b09297a-af93-4b3a-9870-ef5603ce94fd","Type":"ContainerDied","Data":"1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171"}
Sep 17 19:31:24.361532 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.361508    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:31:24.366849 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.366824    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:24.506135 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.506046    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-rbac-proxy-sar-config\") pod \"6b09297a-af93-4b3a-9870-ef5603ce94fd\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") "
Sep 17 19:31:24.506135 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.506099    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc99v\" (UniqueName: \"kubernetes.io/projected/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-api-access-nc99v\") pod \"6b09297a-af93-4b3a-9870-ef5603ce94fd\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") "
Sep 17 19:31:24.506408 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.506159    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b09297a-af93-4b3a-9870-ef5603ce94fd-proxy-tls\") pod \"6b09297a-af93-4b3a-9870-ef5603ce94fd\" (UID: \"6b09297a-af93-4b3a-9870-ef5603ce94fd\") "
Sep 17 19:31:24.506472 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.506429    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b09297a-af93-4b3a-9870-ef5603ce94fd" (UID: "6b09297a-af93-4b3a-9870-ef5603ce94fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:31:24.508229 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.508197    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b09297a-af93-4b3a-9870-ef5603ce94fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b09297a-af93-4b3a-9870-ef5603ce94fd" (UID: "6b09297a-af93-4b3a-9870-ef5603ce94fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:31:24.508348 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.508328    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-api-access-nc99v" (OuterVolumeSpecName: "kube-api-access-nc99v") pod "6b09297a-af93-4b3a-9870-ef5603ce94fd" (UID: "6b09297a-af93-4b3a-9870-ef5603ce94fd"). InnerVolumeSpecName "kube-api-access-nc99v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:31:24.607131 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.607094    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc99v\" (UniqueName: \"kubernetes.io/projected/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-api-access-nc99v\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:31:24.607131 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.607124    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b09297a-af93-4b3a-9870-ef5603ce94fd-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:31:24.607131 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:24.607134    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b09297a-af93-4b3a-9870-ef5603ce94fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:31:25.189840 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.189751    2557 generic.go:356] "Generic (PLEG): container finished" podID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerID="d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c" exitCode=0
Sep 17 19:31:25.189840 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.189804    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"
Sep 17 19:31:25.190051 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.189835    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" event={"ID":"6b09297a-af93-4b3a-9870-ef5603ce94fd","Type":"ContainerDied","Data":"d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c"}
Sep 17 19:31:25.190051 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.189894    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9" event={"ID":"6b09297a-af93-4b3a-9870-ef5603ce94fd","Type":"ContainerDied","Data":"747a994254dba174e64d05e7ed12aa9f167635c88f041aa6d81e14fd6da50fe8"}
Sep 17 19:31:25.190051 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.189922    2557 scope.go:117] "RemoveContainer" containerID="1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171"
Sep 17 19:31:25.197960 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.197939    2557 scope.go:117] "RemoveContainer" containerID="d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c"
Sep 17 19:31:25.205064 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.205043    2557 scope.go:117] "RemoveContainer" containerID="1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171"
Sep 17 19:31:25.205386 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:31:25.205367    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171\": container with ID starting with 1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171 not found: ID does not exist" containerID="1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171"
Sep 17 19:31:25.205444 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.205393    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171"} err="failed to get container status \"1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171\": rpc error: code = NotFound desc = could not find container \"1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171\": container with ID starting with 1a5c2f626501d38df20c65f2254aa52be0fb3664fcd5a9f6608edcb955b4c171 not found: ID does not exist"
Sep 17 19:31:25.205444 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.205413    2557 scope.go:117] "RemoveContainer" containerID="d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c"
Sep 17 19:31:25.205662 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:31:25.205645    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c\": container with ID starting with d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c not found: ID does not exist" containerID="d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c"
Sep 17 19:31:25.205705 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.205666    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c"} err="failed to get container status \"d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c\": rpc error: code = NotFound desc = could not find container \"d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c\": container with ID starting with d7cd5defb3d8e20af2145f94529f341b6a80fcc1e287ff6c2cc654d95c02858c not found: ID does not exist"
Sep 17 19:31:25.209324 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.209303    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"]
Sep 17 19:31:25.213590 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:25.213570    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e2192-predictor-79886769c8-v9dc9"]
Sep 17 19:31:26.834592 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:26.834539    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" path="/var/lib/kubelet/pods/6b09297a-af93-4b3a-9870-ef5603ce94fd/volumes"
Sep 17 19:31:29.367106 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:29.367070    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:34.367064 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:34.367026    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:34.367500 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:34.367119    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:31:39.367100 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:39.367063    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:40.109000 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:40.108964    2557 ???:1] "http2: server: error reading preface from client 10.0.136.140:37248: read tcp 10.0.139.5:10250->10.0.136.140:37248: read: connection reset by peer"
Sep 17 19:31:40.112479 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:40.112460    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:41.036375 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:41.036336    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:41.925854 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:41.925828    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:42.779919 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:42.779877    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:43.650229 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:43.650182    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:44.367036 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:44.367000    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:44.541788 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:44.541758    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:45.415106 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:45.415076    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:46.289122 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:46.289096    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:47.191466 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:47.191433    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:48.083958 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:48.083921    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:48.962738 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:48.962707    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:49.367012 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:49.366974    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:31:49.907380 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:49.907351    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:50.871011 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:50.870982    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:51.426167 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:51.426145    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:31:51.519048 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:51.519016    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07c1ede4-84f8-48db-9888-2b93d021dc55-openshift-service-ca-bundle\") pod \"07c1ede4-84f8-48db-9888-2b93d021dc55\" (UID: \"07c1ede4-84f8-48db-9888-2b93d021dc55\") "
Sep 17 19:31:51.519231 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:51.519081    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c1ede4-84f8-48db-9888-2b93d021dc55-proxy-tls\") pod \"07c1ede4-84f8-48db-9888-2b93d021dc55\" (UID: \"07c1ede4-84f8-48db-9888-2b93d021dc55\") "
Sep 17 19:31:51.519374 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:51.519351    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c1ede4-84f8-48db-9888-2b93d021dc55-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "07c1ede4-84f8-48db-9888-2b93d021dc55" (UID: "07c1ede4-84f8-48db-9888-2b93d021dc55"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:31:51.521016 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:51.520995    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c1ede4-84f8-48db-9888-2b93d021dc55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07c1ede4-84f8-48db-9888-2b93d021dc55" (UID: "07c1ede4-84f8-48db-9888-2b93d021dc55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:31:51.619691 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:51.619660    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07c1ede4-84f8-48db-9888-2b93d021dc55-proxy-tls\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:31:51.619691 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:51.619689    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07c1ede4-84f8-48db-9888-2b93d021dc55-openshift-service-ca-bundle\") on node \"ip-10-0-139-5.ec2.internal\" DevicePath \"\""
Sep 17 19:31:51.826801 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:51.826772    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-e2192-7f99675849-4dd6h_07c1ede4-84f8-48db-9888-2b93d021dc55/switch-graph-e2192/0.log"
Sep 17 19:31:52.291006 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.290975    2557 generic.go:356] "Generic (PLEG): container finished" podID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerID="5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d" exitCode=0
Sep 17 19:31:52.291542 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.291022    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"
Sep 17 19:31:52.291542 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.291069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" event={"ID":"07c1ede4-84f8-48db-9888-2b93d021dc55","Type":"ContainerDied","Data":"5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d"}
Sep 17 19:31:52.291542 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.291134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h" event={"ID":"07c1ede4-84f8-48db-9888-2b93d021dc55","Type":"ContainerDied","Data":"2fe03fbee97c5a250a5d8dc9a7855abb1689912f58b3d7c182c5db3056962c6c"}
Sep 17 19:31:52.291542 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.291168    2557 scope.go:117] "RemoveContainer" containerID="5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d"
Sep 17 19:31:52.300068 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.299886    2557 scope.go:117] "RemoveContainer" containerID="5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d"
Sep 17 19:31:52.300167 ip-10-0-139-5 kubenswrapper[2557]: E0917 19:31:52.300148    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d\": container with ID starting with 5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d not found: ID does not exist" containerID="5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d"
Sep 17 19:31:52.300229 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.300177    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d"} err="failed to get container status \"5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d\": rpc error: code = NotFound desc = could not find container \"5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d\": container with ID starting with 5fd553324a58880d1a1075f3c1bb11851c4087d8179e1495426497eecfb2a44d not found: ID does not exist"
Sep 17 19:31:52.313045 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.313022    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"]
Sep 17 19:31:52.317879 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.317858    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-e2192-7f99675849-4dd6h"]
Sep 17 19:31:52.839873 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:52.839838    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" path="/var/lib/kubelet/pods/07c1ede4-84f8-48db-9888-2b93d021dc55/volumes"
Sep 17 19:31:57.943494 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:57.943462    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-thgrv_fc00a436-36cb-41da-b35c-c63466007e96/global-pull-secret-syncer/0.log"
Sep 17 19:31:58.119658 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:58.119631    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ns2kp_3b22214b-faa3-490b-8e11-2ea7b4e6e15f/konnectivity-agent/0.log"
Sep 17 19:31:58.198074 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:31:58.197996    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-5.ec2.internal_0655c714803c176a16cc58f0c88d6bb7/haproxy/0.log"
Sep 17 19:32:02.823342 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:02.823315    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lkpgf_01856e90-5b5e-4eae-a91d-52466f206b12/node-exporter/0.log"
Sep 17 19:32:02.846763 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:02.846741    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lkpgf_01856e90-5b5e-4eae-a91d-52466f206b12/kube-rbac-proxy/0.log"
Sep 17 19:32:02.871247 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:02.871205    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lkpgf_01856e90-5b5e-4eae-a91d-52466f206b12/init-textfile/0.log"
Sep 17 19:32:04.901675 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:04.901650    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-8724f_49efff88-39af-448c-b06e-4d773e728dc6/networking-console-plugin/0.log"
Sep 17 19:32:05.457291 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:05.457266    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/1.log"
Sep 17 19:32:05.462786 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:05.462762    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-pnx6j_c74d2858-9009-45c9-b7c5-e2ade2bb6311/console-operator/2.log"
Sep 17 19:32:06.840109 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840081    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"]
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840371    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kube-rbac-proxy"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840382    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kube-rbac-proxy"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840403    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840409    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840420    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840427    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840433    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840438    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840448    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840454    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840460    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kube-rbac-proxy"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840465    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kube-rbac-proxy"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840509    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kserve-container"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840516    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb16c91e-ff28-4218-9530-d93855483151" containerName="splitter-graph-46c43"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840523    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b09297a-af93-4b3a-9870-ef5603ce94fd" containerName="kube-rbac-proxy"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840529    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="07c1ede4-84f8-48db-9888-2b93d021dc55" containerName="switch-graph-e2192"
Sep 17 19:32:06.840526 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840534    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kserve-container"
Sep 17 19:32:06.841101 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.840541    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8e63db-202a-448e-95ad-e27b3cd6e310" containerName="kube-rbac-proxy"
Sep 17 19:32:06.843701 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.843684    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:06.846618 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.846599    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sms2k\"/\"kube-root-ca.crt\""
Sep 17 19:32:06.846755 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.846653    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sms2k\"/\"openshift-service-ca.crt\""
Sep 17 19:32:06.848196 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.848180    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sms2k\"/\"default-dockercfg-2v6r8\""
Sep 17 19:32:06.853051 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.853032    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"]
Sep 17 19:32:06.928289 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.928259    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-proc\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:06.928289 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.928288    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-podres\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:06.928509 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.928307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-lib-modules\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:06.928509 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.928346    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-sys\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:06.928509 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:06.928408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjztp\" (UniqueName: \"kubernetes.io/projected/261d8692-8a25-4e3e-b367-58950b4efc46-kube-api-access-rjztp\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.028820 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.028791    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-sys\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.028820 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.028825    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjztp\" (UniqueName: \"kubernetes.io/projected/261d8692-8a25-4e3e-b367-58950b4efc46-kube-api-access-rjztp\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.029059 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.028874    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-proc\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.029059 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.028889    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-podres\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.029059 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.028904    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-lib-modules\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.029059 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.028912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-sys\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.029059 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.028992    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-proc\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.029059 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.029009    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-podres\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.029059 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.029017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/261d8692-8a25-4e3e-b367-58950b4efc46-lib-modules\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.038376 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.038346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjztp\" (UniqueName: \"kubernetes.io/projected/261d8692-8a25-4e3e-b367-58950b4efc46-kube-api-access-rjztp\") pod \"perf-node-gather-daemonset-gzqsp\" (UID: \"261d8692-8a25-4e3e-b367-58950b4efc46\") " pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.153387 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.153309    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:07.423430 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.423405    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"]
Sep 17 19:32:07.425862 ip-10-0-139-5 kubenswrapper[2557]: W0917 19:32:07.425838    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod261d8692_8a25_4e3e_b367_58950b4efc46.slice/crio-8e182364a93c348facd92b017fdaa077d229988156913d20ca4b4461bbb7be0f WatchSource:0}: Error finding container 8e182364a93c348facd92b017fdaa077d229988156913d20ca4b4461bbb7be0f: Status 404 returned error can't find the container with id 8e182364a93c348facd92b017fdaa077d229988156913d20ca4b4461bbb7be0f
Sep 17 19:32:07.532339 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.532317    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4xzv_4494cc42-c46b-47e9-a716-42b1777a1e75/dns/0.log"
Sep 17 19:32:07.557119 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.557097    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4xzv_4494cc42-c46b-47e9-a716-42b1777a1e75/kube-rbac-proxy/0.log"
Sep 17 19:32:07.637224 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:07.637197    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8gdkz_4a59040e-3d81-4961-8cd1-8fc477b7470e/dns-node-resolver/0.log"
Sep 17 19:32:08.223868 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:08.223842    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4947k_f1b9b717-c3af-4187-b781-b6145b367908/node-ca/0.log"
Sep 17 19:32:08.353059 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:08.353020    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp" event={"ID":"261d8692-8a25-4e3e-b367-58950b4efc46","Type":"ContainerStarted","Data":"6943879710a5e031fe6c6a34bc261972417732708f7bde7498893bb5a7c9a56c"}
Sep 17 19:32:08.353257 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:08.353069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp" event={"ID":"261d8692-8a25-4e3e-b367-58950b4efc46","Type":"ContainerStarted","Data":"8e182364a93c348facd92b017fdaa077d229988156913d20ca4b4461bbb7be0f"}
Sep 17 19:32:08.353257 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:08.353100    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:08.375396 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:08.375353    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp" podStartSLOduration=2.375336818 podStartE2EDuration="2.375336818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:32:08.373464793 +0000 UTC m=+3201.728723566" watchObservedRunningTime="2026-09-17 19:32:08.375336818 +0000 UTC m=+3201.730595590"
Sep 17 19:32:09.462403 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:09.462375    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zzl8m_c93cb7fc-25e5-42fe-8050-7079316270d5/serve-healthcheck-canary/0.log"
Sep 17 19:32:09.984876 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:09.984848    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cbb2x_2e6d3062-49a3-4f42-9ace-8ba0d7c90f16/kube-rbac-proxy/0.log"
Sep 17 19:32:10.006196 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:10.006167    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cbb2x_2e6d3062-49a3-4f42-9ace-8ba0d7c90f16/exporter/0.log"
Sep 17 19:32:10.027826 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:10.027801    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cbb2x_2e6d3062-49a3-4f42-9ace-8ba0d7c90f16/extractor/0.log"
Sep 17 19:32:12.481382 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:12.481352    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-j52ld_d88c8014-29b0-44f2-8fa0-854dce39b671/server/0.log"
Sep 17 19:32:12.572009 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:12.571978    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-l64kh_8a7204ed-f1a1-4d31-9c0c-74b9b14609ef/manager/0.log"
Sep 17 19:32:12.651733 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:12.651702    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-chjq2_6bccb4ef-e450-4e8d-b30f-f56a3754b53e/seaweedfs/0.log"
Sep 17 19:32:14.366665 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:14.366641    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sms2k/perf-node-gather-daemonset-gzqsp"
Sep 17 19:32:16.981311 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:16.981278    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-vv6wk_5f273124-4276-4588-962a-4424f23b76b8/kube-storage-version-migrator-operator/1.log"
Sep 17 19:32:16.982110 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:16.982092    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-vv6wk_5f273124-4276-4588-962a-4424f23b76b8/kube-storage-version-migrator-operator/0.log"
Sep 17 19:32:18.551157 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.551126    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wl4cz_9bbd018c-8b15-4a81-a4b6-b96685c1afff/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:32:18.575363 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.575332    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wl4cz_9bbd018c-8b15-4a81-a4b6-b96685c1afff/egress-router-binary-copy/0.log"
Sep 17 19:32:18.598408 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.598385    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wl4cz_9bbd018c-8b15-4a81-a4b6-b96685c1afff/cni-plugins/0.log"
Sep 17 19:32:18.621994 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.621969    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wl4cz_9bbd018c-8b15-4a81-a4b6-b96685c1afff/bond-cni-plugin/0.log"
Sep 17 19:32:18.648162 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.648140    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wl4cz_9bbd018c-8b15-4a81-a4b6-b96685c1afff/routeoverride-cni/0.log"
Sep 17 19:32:18.673035 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.673007    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wl4cz_9bbd018c-8b15-4a81-a4b6-b96685c1afff/whereabouts-cni-bincopy/0.log"
Sep 17 19:32:18.707925 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.707904    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wl4cz_9bbd018c-8b15-4a81-a4b6-b96685c1afff/whereabouts-cni/0.log"
Sep 17 19:32:18.832085 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.832060    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lbftv_b92bcb7b-6ffd-4148-897b-eb21b6ed05df/kube-multus/0.log"
Sep 17 19:32:18.943713 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.943685    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qvr8b_a0f20d3b-b7f6-435b-9375-bef4013e5de5/network-metrics-daemon/0.log"
Sep 17 19:32:18.967135 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:18.967110    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qvr8b_a0f20d3b-b7f6-435b-9375-bef4013e5de5/kube-rbac-proxy/0.log"
Sep 17 19:32:20.238395 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:20.238371    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-controller/0.log"
Sep 17 19:32:20.258885 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:20.258849    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/1.log"
Sep 17 19:32:20.259033 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:20.258960    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovn-acl-logging/0.log"
Sep 17 19:32:20.279668 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:20.279642    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/kube-rbac-proxy-node/0.log"
Sep 17 19:32:20.302589 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:20.302567    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:32:20.324300 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:20.324280    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/northd/0.log"
Sep 17 19:32:20.347505 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:20.347482    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/nbdb/0.log"
Sep 17 19:32:20.371687 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:20.371668    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/sbdb/0.log"
Sep 17 19:32:20.411653 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:20.411625    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z6rd7_98765f76-dc15-47c1-ab00-28533582d24b/ovnkube-controller/0.log"
Sep 17 19:32:21.387697 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:21.387668    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-m7l6f_38340984-1048-4349-93ae-211944b75b16/check-endpoints/0.log"
Sep 17 19:32:21.471681 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:21.471652    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-x7wf6_19f4f176-cbe7-4e1b-bb05-d7483eefe906/network-check-target-container/0.log"
Sep 17 19:32:22.486796 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:22.486770    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-7wll2_2fb1140a-d7ef-4ac7-bb63-17106262554a/iptables-alerter/0.log"
Sep 17 19:32:23.235178 ip-10-0-139-5 kubenswrapper[2557]: I0917 19:32:23.235153    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-jnmf8_8b2ec2ed-cfcb-4f35-8195-ccb69140aa71/tuned/0.log"