Sep 17 23:08:38.664662 ip-10-0-142-96 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 23:08:38.664676 ip-10-0-142-96 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 23:08:38.664688 ip-10-0-142-96 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 23:08:38.665057 ip-10-0-142-96 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 23:08:48.906413 ip-10-0-142-96 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 23:08:48.906431 ip-10-0-142-96 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2a856932551d4b5da42927d056fa92b6 --
Sep 17 23:10:35.709345 ip-10-0-142-96 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:10:38.931638 ip-10-0-142-96 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:38.931638 ip-10-0-142-96 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:10:38.931638 ip-10-0-142-96 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:38.931638 ip-10-0-142-96 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:10:38.931638 ip-10-0-142-96 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:38.932602 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.931724    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:10:38.936570 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936541    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:38.936570 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936563    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:38.936570 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936567    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:38.936570 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936570    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:38.936570 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936573    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:38.936570 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936577    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:38.936570 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936579    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:38.936570 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936582    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:38.936570 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936586    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936589    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936592    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936595    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936598    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936601    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936604    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936606    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936609    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936612    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936614    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936617    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936619    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936623    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936626    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936628    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936632    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936635    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936637    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936640    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:38.936898 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936642    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936645    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936648    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936650    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936653    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936655    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936658    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936660    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936663    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936665    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936668    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936670    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936673    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936675    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936679    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936683    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936687    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936690    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936692    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:38.937418 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936695    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936698    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936701    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936703    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936706    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936709    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936713    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936717    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936720    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936724    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936727    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936730    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936733    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936735    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936738    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936742    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936744    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936747    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936751    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:38.937936 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936754    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936756    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936759    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936762    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936764    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936767    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936769    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936772    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936776    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936779    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936782    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936784    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936787    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936789    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936792    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936795    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936797    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936802    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936807    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936810    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:38.938423 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936813    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936816    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936819    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936821    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936824    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936826    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936830    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936833    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936836    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936839    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936842    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936845    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.936847    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937611    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937619    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937622    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937626    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937629    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937632    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:38.938921 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937637    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937641    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937644    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937647    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937650    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937653    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937656    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937659    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937661    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937664    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937667    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937670    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937673    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937676    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937679    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937682    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937684    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937687    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937689    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937692    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:38.939411 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937695    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937698    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937701    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937704    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937706    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937709    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937712    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937714    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937717    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937720    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937723    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937726    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937728    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937731    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937734    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937738    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937741    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937745    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937748    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:38.939954 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937750    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937753    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937755    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937758    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937760    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937763    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937766    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937768    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937771    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937773    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937775    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937778    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937781    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937783    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937786    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937789    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937791    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937794    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937797    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937799    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:38.940469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937802    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937805    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937807    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937811    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937813    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937816    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937819    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937821    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937824    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937826    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937829    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937832    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937834    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937838    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937840    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937843    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937845    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937848    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937860    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937863    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:38.940977 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937866    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937868    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937871    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937874    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937876    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937879    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937882    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937885    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937888    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937890    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937893    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937896    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937900    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.937903    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938327    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938332    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938335    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938338    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938341    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938344    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:38.941499 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938347    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938350    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938352    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938356    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938359    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938362    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938364    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938367    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938369    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938372    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938375    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938378    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938381    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938383    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938386    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938389    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938391    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938394    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938396    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938398    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:38.942028 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938401    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938403    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938406    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938408    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938411    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938414    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938417    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938420    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938425    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938429    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938432    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938435    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938438    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938441    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938447    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938451    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938454    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938457    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938460    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:38.942523 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938462    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938465    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938468    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938471    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938474    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938476    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938479    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938482    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938484    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938487    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938490    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938492    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938495    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938497    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938500    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938502    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938505    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938508    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938510    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938513    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:38.942990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938516    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938519    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938522    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938526    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938530    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938533    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938535    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938540    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938543    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938546    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938548    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938550    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938553    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938560    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938564    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938567    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938570    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938574    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938577    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:38.943529 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938580    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938582    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938585    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938588    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938590    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938593    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938595    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938598    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938601    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938603    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938606    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938608    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938611    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938614    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.938616    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938693    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938740    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938747    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938752    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938756    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938761    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:10:38.943990 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938766    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938770    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938773    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938777    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938780    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938784    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938787    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938790    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938793    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938796    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938801    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938804    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938808    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938811    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938814    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938817    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938820    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938825    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938828    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938831    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938834    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938837    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938841    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938844    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:10:38.944562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938848    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938851    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938855    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938858    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938861    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938864    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938867    2583 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938872    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938877    2583 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938881    2583 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938884    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938887    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938890    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938894    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938897    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938900    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938903    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938906    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938909    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938912    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938915    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938918    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938921    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938924    2583 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938929    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:10:38.945185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938932    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938936    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938939    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938942    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938945    2583 flags.go:64] FLAG: --help="false"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938948    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-142-96.ec2.internal"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938953    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938957    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938960    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938963    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938967    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938970    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938973    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938976    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938979    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938982    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938986    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938989    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938991    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938994    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.938998    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939000    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939003    2583 flags.go:64] FLAG: --lock-file=""
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939006    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:10:38.945855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939009    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939012    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939030    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939034    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939037    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939040    2583 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939044    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939047    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939050    2583 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939053    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939057    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939060    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939064    2583 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939067    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939072    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939075    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939078    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939081    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939084    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939087    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939095    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939099    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939102    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:10:38.946501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939105    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939108    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939114    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939117    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939120    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939123    2583 flags.go:64] FLAG: --port="10250"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939126    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939130    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03a92bf77254438c2"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939133    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939136    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939139    2583 flags.go:64] FLAG: --register-node="true"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939142    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939146    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939150    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939153    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939156    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939160    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939163    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939166    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939169    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939172    2583 flags.go:64] FLAG: --runonce="false"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939175    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939179    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939182    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939187    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:10:38.947087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939190    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939193    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939196    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939200    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939203    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939206    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939209    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939212    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939215    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939218    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939221    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939227    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939229    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939232    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939238    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939240    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939243    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939247    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939249    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939253    2583 flags.go:64] FLAG: --v="2"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939257    2583 flags.go:64] FLAG: --version="false"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939262    2583 flags.go:64] FLAG: --vmodule=""
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939266    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939269    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939385    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:38.947707 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939389    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939392    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939395    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939398    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939400    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939403    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939407    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939410    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939413    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939416    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939419    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939421    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939424    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939427    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939429    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939433    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939435    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939438    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939440    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939443    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:38.948405 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939445    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939448    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939451    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939454    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939456    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939459    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939462    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939465    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939467    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939470    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939473    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939476    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939478    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939482    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939485    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939488    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939492    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939495    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939499    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:38.948913 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939502    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939505    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939507    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939510    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939512    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939515    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939518    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939520    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939523    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939526    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939528    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939531    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939533    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939536    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939538    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939541    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939544    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939547    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939550    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939553    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:38.949416 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939555    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939558    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939560    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939564    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939566    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939569    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939573    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939576    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939578    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939581    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939583    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939587    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939591    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939593    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939598    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939601    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939604    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939607    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939610    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:38.949928 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939613    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939616    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939619    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939622    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939625    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939627    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939630    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939633    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939635    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939638    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939642    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939645    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939649    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939652    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939655    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939658    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939660    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939665    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939668    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:38.950422 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.939672    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.939678    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.947239    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.947268    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.947291    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.947298    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947370    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947377    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947380    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947383    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947386    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947390    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947393    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947395    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947399    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:38.950891 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947402    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947405    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947408    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947410    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947413    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947416    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947418    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947421    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947424    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947426    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947429    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947431    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947434    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947437    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947440    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947442    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947445    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947447    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947450    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947453    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:38.951295 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947457    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947460    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947463    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947466    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947468    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947471    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947474    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947476    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947479    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947482    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947484    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947487    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947490    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947492    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947495    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947498    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947501    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947503    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947506    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947509    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:38.951875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947511    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947514    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947517    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947519    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947522    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947524    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947528    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947532    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947535    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947538    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947541    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947544    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947548    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947551    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947554    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947556    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947559    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947561    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947564    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:38.952634 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947566    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947569    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947571    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947574    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947577    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947580    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947582    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947585    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947587    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947590    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947592    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947595    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947597    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947601    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947603    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947606    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947610    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947614    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947617    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:38.953222 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947620    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947623    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947626    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947629    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947631    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947634    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947637    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947640    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947643    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947646    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947649    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947652    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.947658    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947778    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:38.953995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947782    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947785    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947789    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947793    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947796    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947799    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947802    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947804    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947807    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947811    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947813    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947816    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947819    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947822    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947824    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947827    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947829    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947831    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947835    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:38.954512 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947839    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947842    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947845    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947847    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947850    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947853    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947856    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947859    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947862    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947865    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947868    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947870    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947873    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947876    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947878    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947881    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947883    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947886    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947889    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947891    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:38.954989 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947894    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947896    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947899    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947902    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947905    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947907    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947910    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947912    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947915    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947917    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947920    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947923    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947926    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947928    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947931    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947933    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947936    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947939    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947941    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947944    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:38.955554 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947947    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947950    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947953    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947956    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947958    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947961    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947965    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947968    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947971    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947974    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947977    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947979    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947983    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947985    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947989    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947992    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947994    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.947997    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948000    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:38.956202 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948003    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948006    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948009    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948011    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948014    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948017    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948037    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948039    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948042    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948045    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948048    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948051    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948054    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948056    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948059    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948064    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948067    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948070    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948072    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:38.956711 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:38.948075    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:38.957633 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.948080    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:38.957633 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.948218    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:10:38.957633 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.951274    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:10:38.957633 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.952151    2583 server.go:1019] "Starting client certificate rotation"
Sep 17 23:10:38.957633 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.952276    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:38.957633 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.952335    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:38.957633 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.954448    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:38.957633 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.955646    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:38.969033 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.968997    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:10:38.971519 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.971496    2583 log.go:25] "Validated CRI v1 image API"
Sep 17 23:10:38.975066 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.975040    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:10:38.977256 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.977233    2583 fs.go:135] Filesystem UUIDs: map[0ff1716f-f47d-4e48-afb3-8d2440a3540c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7b3c9351-42f7-4994-8337-fb8369a15a3c:/dev/nvme0n1p3]
Sep 17 23:10:38.977316 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.977256    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:10:38.981803 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.981678    2583 manager.go:216] Machine: {Timestamp:2026-09-17 23:10:38.981176144 +0000 UTC m=+1.915621885 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110689 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dbe0d67e84d81f1424627cf9fb4f5 SystemUUID:ec2dbe0d-67e8-4d81-f142-4627cf9fb4f5 BootID:2a856932-551d-4b5d-a429-27d056fa92b6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:e1:ee:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:e1:ee:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:38:dc:bf:73:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:10:38.981803 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.981788    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:10:38.981968 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.981875    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:10:38.983422 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.983389    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:10:38.983578 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.983424    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:10:38.984683 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.984670    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:10:38.984735 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.984688    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:10:38.984735 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.984696    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:38.984735 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.984708    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:38.984814 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.984790    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:10:38.988774 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.988756    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:38.988903 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.988892    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:10:38.988959 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.988937    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:10:38.988959 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.988952    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:10:38.989081 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.988977    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:10:38.989081 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.988987    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:10:38.989081 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.988996    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:10:38.989987 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.989961    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:38.990096 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.989999    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:38.991360 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.991339    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:38.991957 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.991943    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:10:38.995460 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.995440    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:10:38.995544 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.995472    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:10:38.996601 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996588    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:10:38.996640 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996605    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:10:38.996640 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996612    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:10:38.996640 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996626    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:10:38.996640 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996636    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:10:38.996838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996642    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:10:38.996838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996656    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:10:38.996838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996661    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:10:38.996838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996667    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:10:38.996838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996673    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:10:38.996838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996685    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:10:38.996838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996690    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:10:38.996838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996717    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:10:38.996838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.996722    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:10:38.999554 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:38.999530    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:10:38.999670 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:38.999567    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:10:38.999670 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:38.999650    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:10:39.000947 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.000925    2583 server.go:1268] "Started kubelet"
Sep 17 23:10:39.001041 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.001000    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:10:39.001082 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.001035    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:10:39.001122 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.001103    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:10:39.026711 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.026665    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:10:39.026761 ip-10-0-142-96 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:10:39.029712 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.028502    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:10:39.029712 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.029070    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:10:39.034064 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.033232    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-96.ec2.internal.18d63e8418e947b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-96.ec2.internal,UID:ip-10-0-142-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-96.ec2.internal,},FirstTimestamp:2026-09-17 23:10:39.000897463 +0000 UTC m=+1.935343230,LastTimestamp:2026-09-17 23:10:39.000897463 +0000 UTC m=+1.935343230,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-96.ec2.internal,}"
Sep 17 23:10:39.035943 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.035927    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:10:39.036002 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.035942    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:39.036162 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.036123    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-96.ec2.internal\" not found"
Sep 17 23:10:39.036572 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.036552    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:10:39.036675 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.036624    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:10:39.036675 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.036657    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:10:39.036822 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.036806    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:10:39.036822 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.036819    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:10:39.037477 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.037286    2583 factory.go:55] Registering systemd factory
Sep 17 23:10:39.037477 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.037302    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:10:39.037729 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.037711    2583 factory.go:153] Registering CRI-O factory
Sep 17 23:10:39.037779 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.037733    2583 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:10:39.037835 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.037794    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:10:39.037835 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.037816    2583 factory.go:103] Registering Raw factory
Sep 17 23:10:39.037835 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.037831    2583 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:10:39.038767 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.038752    2583 manager.go:349] Starting recovery of all containers
Sep 17 23:10:39.048354 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.048298    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:10:39.048533 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.048506    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:10:39.049279 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.049265    2583 manager.go:354] Recovery completed
Sep 17 23:10:39.053190 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.053174    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.054988 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.054961    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.055065 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.055005    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.055065 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.055034    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.055658 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.055639    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:10:39.055658 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.055652    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:10:39.055829 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.055664    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.055829 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.055676    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:39.056788 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.056772    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.056860 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.056801    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.056860 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.056814    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.058035 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.057998    2583 policy_none.go:49] "None policy: Start"
Sep 17 23:10:39.058181 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.058016    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:10:39.058220 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.058188    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:10:39.060240 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.060227    2583 policy_none.go:47] "Start"
Sep 17 23:10:39.064486 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.064395    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-96.ec2.internal.18d63e841c22a4d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-96.ec2.internal,UID:ip-10-0-142-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-96.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-96.ec2.internal,},FirstTimestamp:2026-09-17 23:10:39.054988502 +0000 UTC m=+1.989434243,LastTimestamp:2026-09-17 23:10:39.054988502 +0000 UTC m=+1.989434243,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-96.ec2.internal,}"
Sep 17 23:10:39.070645 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.070559    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-96.ec2.internal.18d63e841c230089  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-96.ec2.internal,UID:ip-10-0-142-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-96.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-96.ec2.internal,},FirstTimestamp:2026-09-17 23:10:39.055011977 +0000 UTC m=+1.989457717,LastTimestamp:2026-09-17 23:10:39.055011977 +0000 UTC m=+1.989457717,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-96.ec2.internal,}"
Sep 17 23:10:39.075675 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.075642    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ctt7d"
Sep 17 23:10:39.075806 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.075601    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-96.ec2.internal.18d63e841c23718e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-96.ec2.internal,UID:ip-10-0-142-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-96.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-96.ec2.internal,},FirstTimestamp:2026-09-17 23:10:39.05504091 +0000 UTC m=+1.989486651,LastTimestamp:2026-09-17 23:10:39.05504091 +0000 UTC m=+1.989486651,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-96.ec2.internal,}"
Sep 17 23:10:39.081736 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.081649    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-96.ec2.internal.18d63e841c22a4d6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-96.ec2.internal.18d63e841c22a4d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-96.ec2.internal,UID:ip-10-0-142-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-96.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-96.ec2.internal,},FirstTimestamp:2026-09-17 23:10:39.054988502 +0000 UTC m=+1.989434243,LastTimestamp:2026-09-17 23:10:39.056787375 +0000 UTC m=+1.991233116,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-96.ec2.internal,}"
Sep 17 23:10:39.085516 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.085201    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ctt7d"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.087108    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-96.ec2.internal.18d63e841c230089\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-96.ec2.internal.18d63e841c230089  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-96.ec2.internal,UID:ip-10-0-142-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-96.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-96.ec2.internal,},FirstTimestamp:2026-09-17 23:10:39.055011977 +0000 UTC m=+1.989457717,LastTimestamp:2026-09-17 23:10:39.056806358 +0000 UTC m=+1.991252100,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-96.ec2.internal,}"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.095700    2583 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.095742    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.095755    2583 server.go:87] "Starting device plugin registration server"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.096056    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.096074    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.096166    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.096250    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.096259    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.096806    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:10:39.106129 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.096864    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-96.ec2.internal\" not found"
Sep 17 23:10:39.149551 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.149505    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:10:39.150855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.150826    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:10:39.150989 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.150890    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:10:39.150989 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.150922    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:10:39.150989 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.150966    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:10:39.153511 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.153486    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:39.196864 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.196783    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.197828 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.197809    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.197937 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.197849    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.197937 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.197860    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.197937 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.197885    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.207549 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.207523    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.207549 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.207551    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-96.ec2.internal\": node \"ip-10-0-142-96.ec2.internal\" not found"
Sep 17 23:10:39.251559 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.251521    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal"]
Sep 17 23:10:39.251750 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.251615    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.252559 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.252541    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-96.ec2.internal\" not found"
Sep 17 23:10:39.253315 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.253296    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.253386 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.253328    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.253386 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.253341    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.255757 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.255742    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.255872 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.255857    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.255913 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.255889    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.256628 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.256611    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.256628 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.256623    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.256709 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.256643    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.256709 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.256644    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.256709 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.256654    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.256709 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.256663    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.258984 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.258966    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.259097 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.258998    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.259934 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.259915    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.260056 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.259946    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.260056 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.259956    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.280036 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.280002    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-96.ec2.internal\" not found" node="ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.284941 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.284910    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-96.ec2.internal\" not found" node="ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.339057 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.338987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27ea57f9def2c8b7b50c3726e00d3bc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal\" (UID: \"27ea57f9def2c8b7b50c3726e00d3bc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.339057 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.339053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ea57f9def2c8b7b50c3726e00d3bc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal\" (UID: \"27ea57f9def2c8b7b50c3726e00d3bc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.339265 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.339076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c587c9980c41ab67e0f45c632018ea28-config\") pod \"kube-apiserver-proxy-ip-10-0-142-96.ec2.internal\" (UID: \"c587c9980c41ab67e0f45c632018ea28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.353334 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.353307    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-96.ec2.internal\" not found"
Sep 17 23:10:39.439922 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.439881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27ea57f9def2c8b7b50c3726e00d3bc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal\" (UID: \"27ea57f9def2c8b7b50c3726e00d3bc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.440110 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.439937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ea57f9def2c8b7b50c3726e00d3bc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal\" (UID: \"27ea57f9def2c8b7b50c3726e00d3bc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.440110 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.439962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c587c9980c41ab67e0f45c632018ea28-config\") pod \"kube-apiserver-proxy-ip-10-0-142-96.ec2.internal\" (UID: \"c587c9980c41ab67e0f45c632018ea28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.440110 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.439896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27ea57f9def2c8b7b50c3726e00d3bc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal\" (UID: \"27ea57f9def2c8b7b50c3726e00d3bc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.440110 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.439999    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c587c9980c41ab67e0f45c632018ea28-config\") pod \"kube-apiserver-proxy-ip-10-0-142-96.ec2.internal\" (UID: \"c587c9980c41ab67e0f45c632018ea28\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.440110 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.440047    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ea57f9def2c8b7b50c3726e00d3bc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal\" (UID: \"27ea57f9def2c8b7b50c3726e00d3bc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.454063 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.453979    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-96.ec2.internal\" not found"
Sep 17 23:10:39.554822 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.554779    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-96.ec2.internal\" not found"
Sep 17 23:10:39.583991 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.583948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.588449 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.588423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.655572 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.655369    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-96.ec2.internal\" not found"
Sep 17 23:10:39.756015 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.755932    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-96.ec2.internal\" not found"
Sep 17 23:10:39.846495 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.846470    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:39.937112 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.937074    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.952370 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.952335    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:10:39.952522 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.952431    2583 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.142.96:39268->3.226.246.80:6443: use of closed network connection"
Sep 17 23:10:39.952522 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.952462    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:39.952522 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.952468    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:39.952522 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:39.952479    2583 kubelet.go:3250] "Failed creating a mirror pod" err="unexpected error when reading response body. Please retry. Original error: read tcp 10.0.142.96:39268->3.226.246.80:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.952522 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.952494    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal"
Sep 17 23:10:39.970414 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.970378    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:39.989587 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.989553    2583 apiserver.go:52] "Watching apiserver"
Sep 17 23:10:39.998035 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.997990    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:10:39.998367 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:39.998344    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-pzttd","kube-system/konnectivity-agent-xcwf9","kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal","openshift-multus/multus-7wwwj","openshift-multus/multus-additional-cni-plugins-cnrdw","openshift-network-diagnostics/network-check-target-b224x","openshift-cluster-node-tuning-operator/tuned-hfkwd","openshift-dns/node-resolver-dlnfn","openshift-image-registry/node-ca-x9277","openshift-multus/network-metrics-daemon-pk7p2","openshift-network-operator/iptables-alerter-9fctl","openshift-ovn-kubernetes/ovnkube-node-xp97h"]
Sep 17 23:10:40.000862 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.000834    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.000978 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.000909    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:40.002900 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.002879    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:10:40.005194 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.005173    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.005755 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.005735    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:10:40.005843 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.005820    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:10:40.006313 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.006264    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pwbx9\""
Sep 17 23:10:40.007446 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.007428    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:10:40.007599 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.007581    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gpqtf\""
Sep 17 23:10:40.007760 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.007744    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:10:40.008109 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.008093    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.008248 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.008095    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:10:40.010466 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.010443    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.010571 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.010501    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:10:40.010571 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.010501    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:10:40.010571 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.010542    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:10:40.010727 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.010652    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:10:40.011143 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.011127    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gqxz4\""
Sep 17 23:10:40.012874 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.012858    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rp98k\""
Sep 17 23:10:40.012944 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.012875    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:40.012993 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.012961    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:40.013251 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.013235    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:10:40.013301 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.013249    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:10:40.015333 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.015309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.017513 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.017490    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.019290 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.019271    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:40.019399 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.019337    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gsvdk\""
Sep 17 23:10:40.019399 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.019385    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mhmrf\""
Sep 17 23:10:40.019499 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.019404    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:40.019842 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.019826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.019892 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.019841    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:10:40.019892 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.019859    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:10:40.022475 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.022453    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s2gt5\""
Sep 17 23:10:40.022595 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.022525    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:10:40.022716 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.022699    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:40.022771 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.022750    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:10:40.022771 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.022752    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:10:40.022869 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.022788    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:40.025369 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.025347    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.030653 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.030631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.031093 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.031071    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:40.031467 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.031454    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:10:40.031846 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.031827    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:40.031952 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.031934    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8shpk\""
Sep 17 23:10:40.034690 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.034669    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:10:40.034788 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.034715    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:10:40.036763 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.036743    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:40.037452 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.037433    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:10:40.038396 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.038379    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zchs9\""
Sep 17 23:10:40.038643 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.038608    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:10:40.038718 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.038665    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:10:40.038718 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.038691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:10:40.039063 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.039047    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:10:40.043284 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shkm6\" (UniqueName: \"kubernetes.io/projected/99314a8b-f4ea-4521-ad1f-6d152fd397fb-kube-api-access-shkm6\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.043395 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043299    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-systemd\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.043395 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043330    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-host\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.043395 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppnrk\" (UniqueName: \"kubernetes.io/projected/d9f1f9aa-a2cd-456e-9606-fdb3218caede-kube-api-access-ppnrk\") pod \"iptables-alerter-9fctl\" (UID: \"d9f1f9aa-a2cd-456e-9606-fdb3218caede\") " pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.043395 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-cni-netd\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.043556 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqgx\" (UniqueName: \"kubernetes.io/projected/6ee7a800-e764-412e-9826-9d242b39b039-kube-api-access-5wqgx\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.043556 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-hostroot\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.043556 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:40.043556 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb5aa21c-1af6-47e6-b02e-1417aecdc5ad-host\") pod \"node-ca-x9277\" (UID: \"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad\") " pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.043556 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ee7a800-e764-412e-9826-9d242b39b039-ovn-node-metrics-cert\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-run-k8s-cni-cncf-io\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-var-lib-cni-bin\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-multus-conf-dir\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043704    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-sysconfig\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043734    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9f1f9aa-a2cd-456e-9606-fdb3218caede-iptables-alerter-script\") pod \"iptables-alerter-9fctl\" (UID: \"d9f1f9aa-a2cd-456e-9606-fdb3218caede\") " pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99314a8b-f4ea-4521-ad1f-6d152fd397fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/986816f6-1fb8-429d-a4aa-c4778d601b56-dbus\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-var-lib-cni-multus\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99314a8b-f4ea-4521-ad1f-6d152fd397fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-var-lib-kubelet\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.043887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-run-openvswitch\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ee7a800-e764-412e-9826-9d242b39b039-ovnkube-config\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-run-multus-certs\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.043996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99314a8b-f4ea-4521-ad1f-6d152fd397fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-sysctl-conf\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrwl\" (UniqueName: \"kubernetes.io/projected/c73226c0-ffce-43af-89c5-e7197f34364e-kube-api-access-rhrwl\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044093    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-run-netns\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-log-socket\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-multus-socket-dir-parent\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99314a8b-f4ea-4521-ad1f-6d152fd397fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-run-ovn-kubernetes\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-sys\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044225    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-lib-modules\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.044381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-registration-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044273    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-etc-openvswitch\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79e79ce8-102c-4719-9637-12f50a0382c3-cni-binary-copy\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-run-netns\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99314a8b-f4ea-4521-ad1f-6d152fd397fb-system-cni-dir\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kc98\" (UniqueName: \"kubernetes.io/projected/fb5aa21c-1af6-47e6-b02e-1417aecdc5ad-kube-api-access-5kc98\") pod \"node-ca-x9277\" (UID: \"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad\") " pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-kubelet\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044453    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-multus-cni-dir\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99314a8b-f4ea-4521-ad1f-6d152fd397fb-cnibin\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4ts\" (UniqueName: \"kubernetes.io/projected/1aced041-ad96-4073-8c9a-dc28c37ea129-kube-api-access-bl4ts\") pod \"node-resolver-dlnfn\" (UID: \"1aced041-ad96-4073-8c9a-dc28c37ea129\") " pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044537    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ee7a800-e764-412e-9826-9d242b39b039-ovnkube-script-lib\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drzpr\" (UniqueName: \"kubernetes.io/projected/79e79ce8-102c-4719-9637-12f50a0382c3-kube-api-access-drzpr\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-kubernetes\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9f1f9aa-a2cd-456e-9606-fdb3218caede-host-slash\") pod \"iptables-alerter-9fctl\" (UID: \"d9f1f9aa-a2cd-456e-9606-fdb3218caede\") " pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-systemd-units\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.045083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-node-log\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-var-lib-kubelet\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044750    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-run\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2c12fff-e88e-47a7-95d5-3897d1c5133a-tmp\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aced041-ad96-4073-8c9a-dc28c37ea129-tmp-dir\") pod \"node-resolver-dlnfn\" (UID: \"1aced041-ad96-4073-8c9a-dc28c37ea129\") " pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-var-lib-openvswitch\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-modprobe-d\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-tuned\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-device-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-sys-fs\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5aa21c-1af6-47e6-b02e-1417aecdc5ad-serviceca\") pod \"node-ca-x9277\" (UID: \"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad\") " pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.044970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/986816f6-1fb8-429d-a4aa-c4778d601b56-kubelet-config\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79e79ce8-102c-4719-9637-12f50a0382c3-multus-daemon-config\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-etc-kubernetes\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-sysctl-d\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4e89f46-8753-4be5-be61-9ccbaf154c6f-agent-certs\") pod \"konnectivity-agent-xcwf9\" (UID: \"b4e89f46-8753-4be5-be61-9ccbaf154c6f\") " pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1aced041-ad96-4073-8c9a-dc28c37ea129-hosts-file\") pod \"node-resolver-dlnfn\" (UID: \"1aced041-ad96-4073-8c9a-dc28c37ea129\") " pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.045652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-socket-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqz5c\" (UniqueName: \"kubernetes.io/projected/a3e1aec2-2b62-46a2-8c82-93456e200ba8-kube-api-access-nqz5c\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-run-systemd\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ee7a800-e764-412e-9826-9d242b39b039-env-overrides\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045251    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4e89f46-8753-4be5-be61-9ccbaf154c6f-konnectivity-ca\") pod \"konnectivity-agent-xcwf9\" (UID: \"b4e89f46-8753-4be5-be61-9ccbaf154c6f\") " pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-slash\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-run-ovn\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045356    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-system-cni-dir\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-cnibin\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtcp\" (UniqueName: \"kubernetes.io/projected/d2c12fff-e88e-47a7-95d5-3897d1c5133a-kube-api-access-pwtcp\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.046182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.045434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-cni-bin\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.054420 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.054392    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:40.091266 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.091204    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:05:39 +0000 UTC" deadline="2028-03-21 23:46:12.848058831 +0000 UTC"
Sep 17 23:10:40.091266 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.091262    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13224h35m32.756799915s"
Sep 17 23:10:40.093590 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.093562    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:40.145723 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145676    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-run-k8s-cni-cncf-io\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.145723 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-var-lib-cni-bin\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.145723 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-multus-conf-dir\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.145723 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-sysconfig\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9f1f9aa-a2cd-456e-9606-fdb3218caede-iptables-alerter-script\") pod \"iptables-alerter-9fctl\" (UID: \"d9f1f9aa-a2cd-456e-9606-fdb3218caede\") " pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145835    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-run-k8s-cni-cncf-io\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-sysconfig\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99314a8b-f4ea-4521-ad1f-6d152fd397fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/986816f6-1fb8-429d-a4aa-c4778d601b56-dbus\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-multus-conf-dir\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-var-lib-cni-multus\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.145992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99314a8b-f4ea-4521-ad1f-6d152fd397fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-var-lib-kubelet\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.146058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-run-openvswitch\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ee7a800-e764-412e-9826-9d242b39b039-ovnkube-config\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-var-lib-kubelet\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/986816f6-1fb8-429d-a4aa-c4778d601b56-dbus\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.146252    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-run-openvswitch\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.146365    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret podName:986816f6-1fb8-429d-a4aa-c4778d601b56 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:40.646310806 +0000 UTC m=+3.580756559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret") pod "global-pull-secret-syncer-pzttd" (UID: "986816f6-1fb8-429d-a4aa-c4778d601b56") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146427    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9f1f9aa-a2cd-456e-9606-fdb3218caede-iptables-alerter-script\") pod \"iptables-alerter-9fctl\" (UID: \"d9f1f9aa-a2cd-456e-9606-fdb3218caede\") " pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-var-lib-cni-bin\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/99314a8b-f4ea-4521-ad1f-6d152fd397fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.146550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-var-lib-cni-multus\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146571    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146200    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-run-multus-certs\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146659    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-run-multus-certs\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99314a8b-f4ea-4521-ad1f-6d152fd397fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-sysctl-conf\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrwl\" (UniqueName: \"kubernetes.io/projected/c73226c0-ffce-43af-89c5-e7197f34364e-kube-api-access-rhrwl\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-run-netns\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-log-socket\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-log-socket\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-run-netns\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-sysctl-conf\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-multus-socket-dir-parent\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99314a8b-f4ea-4521-ad1f-6d152fd397fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-run-ovn-kubernetes\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.146989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/99314a8b-f4ea-4521-ad1f-6d152fd397fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.147160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-sys\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-run-ovn-kubernetes\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-sys\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-lib-modules\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147110    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-multus-socket-dir-parent\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-registration-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147150    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ee7a800-e764-412e-9826-9d242b39b039-ovnkube-config\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-etc-openvswitch\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/99314a8b-f4ea-4521-ad1f-6d152fd397fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79e79ce8-102c-4719-9637-12f50a0382c3-cni-binary-copy\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-registration-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-lib-modules\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-run-netns\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147280    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-etc-openvswitch\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.147902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99314a8b-f4ea-4521-ad1f-6d152fd397fb-system-cni-dir\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kc98\" (UniqueName: \"kubernetes.io/projected/fb5aa21c-1af6-47e6-b02e-1417aecdc5ad-kube-api-access-5kc98\") pod \"node-ca-x9277\" (UID: \"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad\") " pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147337    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-run-netns\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/99314a8b-f4ea-4521-ad1f-6d152fd397fb-system-cni-dir\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-kubelet\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-multus-cni-dir\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147441    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-kubelet\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99314a8b-f4ea-4521-ad1f-6d152fd397fb-cnibin\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4ts\" (UniqueName: \"kubernetes.io/projected/1aced041-ad96-4073-8c9a-dc28c37ea129-kube-api-access-bl4ts\") pod \"node-resolver-dlnfn\" (UID: \"1aced041-ad96-4073-8c9a-dc28c37ea129\") " pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ee7a800-e764-412e-9826-9d242b39b039-ovnkube-script-lib\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-multus-cni-dir\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drzpr\" (UniqueName: \"kubernetes.io/projected/79e79ce8-102c-4719-9637-12f50a0382c3-kube-api-access-drzpr\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/99314a8b-f4ea-4521-ad1f-6d152fd397fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147841    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/99314a8b-f4ea-4521-ad1f-6d152fd397fb-cnibin\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-kubernetes\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147949    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9f1f9aa-a2cd-456e-9606-fdb3218caede-host-slash\") pod \"iptables-alerter-9fctl\" (UID: \"d9f1f9aa-a2cd-456e-9606-fdb3218caede\") " pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.147998    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-kubernetes\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.148650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9f1f9aa-a2cd-456e-9606-fdb3218caede-host-slash\") pod \"iptables-alerter-9fctl\" (UID: \"d9f1f9aa-a2cd-456e-9606-fdb3218caede\") " pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-systemd-units\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-node-log\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79e79ce8-102c-4719-9637-12f50a0382c3-cni-binary-copy\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-node-log\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-var-lib-kubelet\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148181    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-run\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-run\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-host-var-lib-kubelet\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-systemd-units\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2c12fff-e88e-47a7-95d5-3897d1c5133a-tmp\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aced041-ad96-4073-8c9a-dc28c37ea129-tmp-dir\") pod \"node-resolver-dlnfn\" (UID: \"1aced041-ad96-4073-8c9a-dc28c37ea129\") " pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-var-lib-openvswitch\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-modprobe-d\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-tuned\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-device-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-sys-fs\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5aa21c-1af6-47e6-b02e-1417aecdc5ad-serviceca\") pod \"node-ca-x9277\" (UID: \"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad\") " pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.149392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/986816f6-1fb8-429d-a4aa-c4778d601b56-kubelet-config\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79e79ce8-102c-4719-9637-12f50a0382c3-multus-daemon-config\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-etc-kubernetes\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-sysctl-d\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4e89f46-8753-4be5-be61-9ccbaf154c6f-agent-certs\") pod \"konnectivity-agent-xcwf9\" (UID: \"b4e89f46-8753-4be5-be61-9ccbaf154c6f\") " pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148739    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1aced041-ad96-4073-8c9a-dc28c37ea129-hosts-file\") pod \"node-resolver-dlnfn\" (UID: \"1aced041-ad96-4073-8c9a-dc28c37ea129\") " pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-socket-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqz5c\" (UniqueName: \"kubernetes.io/projected/a3e1aec2-2b62-46a2-8c82-93456e200ba8-kube-api-access-nqz5c\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-run-systemd\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ee7a800-e764-412e-9826-9d242b39b039-env-overrides\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4e89f46-8753-4be5-be61-9ccbaf154c6f-konnectivity-ca\") pod \"konnectivity-agent-xcwf9\" (UID: \"b4e89f46-8753-4be5-be61-9ccbaf154c6f\") " pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-slash\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-run-ovn\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-system-cni-dir\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-cnibin\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcp\" (UniqueName: \"kubernetes.io/projected/d2c12fff-e88e-47a7-95d5-3897d1c5133a-kube-api-access-pwtcp\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.150188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-cni-bin\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shkm6\" (UniqueName: \"kubernetes.io/projected/99314a8b-f4ea-4521-ad1f-6d152fd397fb-kube-api-access-shkm6\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149244    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-systemd\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-host\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppnrk\" (UniqueName: \"kubernetes.io/projected/d9f1f9aa-a2cd-456e-9606-fdb3218caede-kube-api-access-ppnrk\") pod \"iptables-alerter-9fctl\" (UID: \"d9f1f9aa-a2cd-456e-9606-fdb3218caede\") " pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-cni-netd\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqgx\" (UniqueName: \"kubernetes.io/projected/6ee7a800-e764-412e-9826-9d242b39b039-kube-api-access-5wqgx\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-hostroot\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb5aa21c-1af6-47e6-b02e-1417aecdc5ad-host\") pod \"node-ca-x9277\" (UID: \"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad\") " pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.150945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.149478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ee7a800-e764-412e-9826-9d242b39b039-ovn-node-metrics-cert\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.151423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.148390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ee7a800-e764-412e-9826-9d242b39b039-ovnkube-script-lib\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.151423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-cnibin\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.151423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151134    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-run-ovn\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.151423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-host\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.151423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-cni-netd\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.151423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-systemd\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.151423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151314    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aced041-ad96-4073-8c9a-dc28c37ea129-tmp-dir\") pod \"node-resolver-dlnfn\" (UID: \"1aced041-ad96-4073-8c9a-dc28c37ea129\") " pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.151423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151328    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-system-cni-dir\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.151423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151400    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-etc-kubernetes\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.151861 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151513    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-sysctl-d\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.151861 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.151548    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:40.151861 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1aced041-ad96-4073-8c9a-dc28c37ea129-hosts-file\") pod \"node-resolver-dlnfn\" (UID: \"1aced041-ad96-4073-8c9a-dc28c37ea129\") " pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.151861 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.151633    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs podName:c73226c0-ffce-43af-89c5-e7197f34364e nodeName:}" failed. No retries permitted until 2026-09-17 23:10:40.651610702 +0000 UTC m=+3.586056430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs") pod "network-metrics-daemon-pk7p2" (UID: "c73226c0-ffce-43af-89c5-e7197f34364e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:40.151861 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb5aa21c-1af6-47e6-b02e-1417aecdc5ad-host\") pod \"node-ca-x9277\" (UID: \"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad\") " pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.151861 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ee7a800-e764-412e-9826-9d242b39b039-env-overrides\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.151861 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-slash\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.151861 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151803    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-run-systemd\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.152270 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4e89f46-8753-4be5-be61-9ccbaf154c6f-konnectivity-ca\") pod \"konnectivity-agent-xcwf9\" (UID: \"b4e89f46-8753-4be5-be61-9ccbaf154c6f\") " pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:10:40.152270 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79e79ce8-102c-4719-9637-12f50a0382c3-hostroot\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.152270 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151970    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-host-cni-bin\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.152270 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.152075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-sys-fs\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.152270 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.152175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-modprobe-d\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.152497 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.152334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79e79ce8-102c-4719-9637-12f50a0382c3-multus-daemon-config\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.152497 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.152404    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee7a800-e764-412e-9826-9d242b39b039-var-lib-openvswitch\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.152497 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.152471    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-device-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.152638 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.152524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/986816f6-1fb8-429d-a4aa-c4778d601b56-kubelet-config\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.152638 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.151759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3e1aec2-2b62-46a2-8c82-93456e200ba8-socket-dir\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.152638 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.152522    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9d4zz"
Sep 17 23:10:40.153151 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.152963    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5aa21c-1af6-47e6-b02e-1417aecdc5ad-serviceca\") pod \"node-ca-x9277\" (UID: \"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad\") " pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.153299 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.153269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2c12fff-e88e-47a7-95d5-3897d1c5133a-tmp\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.153721 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.153421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ee7a800-e764-412e-9826-9d242b39b039-ovn-node-metrics-cert\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.154602 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.154579    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4e89f46-8753-4be5-be61-9ccbaf154c6f-agent-certs\") pod \"konnectivity-agent-xcwf9\" (UID: \"b4e89f46-8753-4be5-be61-9ccbaf154c6f\") " pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:10:40.154757 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.154740    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2c12fff-e88e-47a7-95d5-3897d1c5133a-etc-tuned\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.165652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.165625    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9d4zz"
Sep 17 23:10:40.167922 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.167898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrwl\" (UniqueName: \"kubernetes.io/projected/c73226c0-ffce-43af-89c5-e7197f34364e-kube-api-access-rhrwl\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:40.169883 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.169855    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drzpr\" (UniqueName: \"kubernetes.io/projected/79e79ce8-102c-4719-9637-12f50a0382c3-kube-api-access-drzpr\") pod \"multus-7wwwj\" (UID: \"79e79ce8-102c-4719-9637-12f50a0382c3\") " pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.170534 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.170506    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kc98\" (UniqueName: \"kubernetes.io/projected/fb5aa21c-1af6-47e6-b02e-1417aecdc5ad-kube-api-access-5kc98\") pod \"node-ca-x9277\" (UID: \"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad\") " pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.177487 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.177465    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:40.177487 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.177490    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:40.177654 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.177504    2583 projected.go:196] Error preparing data for projected volume kube-api-access-cpr47 for pod openshift-network-diagnostics/network-check-target-b224x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:40.177654 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.177577    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47 podName:be180008-7f40-4e3c-b31a-bba2323922ef nodeName:}" failed. No retries permitted until 2026-09-17 23:10:40.677555661 +0000 UTC m=+3.612001409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cpr47" (UniqueName: "kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47") pod "network-check-target-b224x" (UID: "be180008-7f40-4e3c-b31a-bba2323922ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:40.179648 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.179622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqgx\" (UniqueName: \"kubernetes.io/projected/6ee7a800-e764-412e-9826-9d242b39b039-kube-api-access-5wqgx\") pod \"ovnkube-node-xp97h\" (UID: \"6ee7a800-e764-412e-9826-9d242b39b039\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.181266 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.181236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shkm6\" (UniqueName: \"kubernetes.io/projected/99314a8b-f4ea-4521-ad1f-6d152fd397fb-kube-api-access-shkm6\") pod \"multus-additional-cni-plugins-cnrdw\" (UID: \"99314a8b-f4ea-4521-ad1f-6d152fd397fb\") " pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.181395 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.181275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtcp\" (UniqueName: \"kubernetes.io/projected/d2c12fff-e88e-47a7-95d5-3897d1c5133a-kube-api-access-pwtcp\") pod \"tuned-hfkwd\" (UID: \"d2c12fff-e88e-47a7-95d5-3897d1c5133a\") " pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.187525 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.187492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppnrk\" (UniqueName: \"kubernetes.io/projected/d9f1f9aa-a2cd-456e-9606-fdb3218caede-kube-api-access-ppnrk\") pod \"iptables-alerter-9fctl\" (UID: \"d9f1f9aa-a2cd-456e-9606-fdb3218caede\") " pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.187759 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.187734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqz5c\" (UniqueName: \"kubernetes.io/projected/a3e1aec2-2b62-46a2-8c82-93456e200ba8-kube-api-access-nqz5c\") pod \"aws-ebs-csi-driver-node-w4ftr\" (UID: \"a3e1aec2-2b62-46a2-8c82-93456e200ba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.191788 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.191767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4ts\" (UniqueName: \"kubernetes.io/projected/1aced041-ad96-4073-8c9a-dc28c37ea129-kube-api-access-bl4ts\") pod \"node-resolver-dlnfn\" (UID: \"1aced041-ad96-4073-8c9a-dc28c37ea129\") " pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.220436 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.220395    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ea57f9def2c8b7b50c3726e00d3bc7.slice/crio-851b5ecd3edff7234b1575c3328ba40a97927f4f184d1de8ff54c14734c04931 WatchSource:0}: Error finding container 851b5ecd3edff7234b1575c3328ba40a97927f4f184d1de8ff54c14734c04931: Status 404 returned error can't find the container with id 851b5ecd3edff7234b1575c3328ba40a97927f4f184d1de8ff54c14734c04931
Sep 17 23:10:40.221127 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.221098    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc587c9980c41ab67e0f45c632018ea28.slice/crio-1b9588ee7534b789e1dfad03cb35f6055d784cc336eed76d650244dc6e12c8c8 WatchSource:0}: Error finding container 1b9588ee7534b789e1dfad03cb35f6055d784cc336eed76d650244dc6e12c8c8: Status 404 returned error can't find the container with id 1b9588ee7534b789e1dfad03cb35f6055d784cc336eed76d650244dc6e12c8c8
Sep 17 23:10:40.226035 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.226006    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:10:40.331463 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.331420    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:10:40.341306 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.341270    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e89f46_8753_4be5_be61_9ccbaf154c6f.slice/crio-25b360840f6fa4c4e8a9f26772f8a2ea8bb4ff8eee0f946fba07ae7e4d21122d WatchSource:0}: Error finding container 25b360840f6fa4c4e8a9f26772f8a2ea8bb4ff8eee0f946fba07ae7e4d21122d: Status 404 returned error can't find the container with id 25b360840f6fa4c4e8a9f26772f8a2ea8bb4ff8eee0f946fba07ae7e4d21122d
Sep 17 23:10:40.342230 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.342210    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr"
Sep 17 23:10:40.351045 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.351009    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e1aec2_2b62_46a2_8c82_93456e200ba8.slice/crio-d689df9a7d945733873ec4b88577006bb61b5792085dc63bf88dc576db314b0f WatchSource:0}: Error finding container d689df9a7d945733873ec4b88577006bb61b5792085dc63bf88dc576db314b0f: Status 404 returned error can't find the container with id d689df9a7d945733873ec4b88577006bb61b5792085dc63bf88dc576db314b0f
Sep 17 23:10:40.368465 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.368436    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7wwwj"
Sep 17 23:10:40.376808 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.376768    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e79ce8_102c_4719_9637_12f50a0382c3.slice/crio-ec97545deaed298d5ad337065f8355065b3e849f45d8a403ddff5d32696206e2 WatchSource:0}: Error finding container ec97545deaed298d5ad337065f8355065b3e849f45d8a403ddff5d32696206e2: Status 404 returned error can't find the container with id ec97545deaed298d5ad337065f8355065b3e849f45d8a403ddff5d32696206e2
Sep 17 23:10:40.386195 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.386160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cnrdw"
Sep 17 23:10:40.387471 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.387432    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:40.397313 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.397282    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99314a8b_f4ea_4521_ad1f_6d152fd397fb.slice/crio-174856722d0d7e1a634061760c2d9e986078f99089f26f1cac6dc8c539b3b2f3 WatchSource:0}: Error finding container 174856722d0d7e1a634061760c2d9e986078f99089f26f1cac6dc8c539b3b2f3: Status 404 returned error can't find the container with id 174856722d0d7e1a634061760c2d9e986078f99089f26f1cac6dc8c539b3b2f3
Sep 17 23:10:40.404600 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.404573    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hfkwd"
Sep 17 23:10:40.413368 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.413340    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c12fff_e88e_47a7_95d5_3897d1c5133a.slice/crio-2a9889921cddef325410b914595ebc774dd53938a3d3763011a5523f84ef79f8 WatchSource:0}: Error finding container 2a9889921cddef325410b914595ebc774dd53938a3d3763011a5523f84ef79f8: Status 404 returned error can't find the container with id 2a9889921cddef325410b914595ebc774dd53938a3d3763011a5523f84ef79f8
Sep 17 23:10:40.426560 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.426529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dlnfn"
Sep 17 23:10:40.435408 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.435379    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aced041_ad96_4073_8c9a_dc28c37ea129.slice/crio-d6c24d65eaf37cb34a05a91dd7cbdabc71a82f7c4613d495f31f016f296307f6 WatchSource:0}: Error finding container d6c24d65eaf37cb34a05a91dd7cbdabc71a82f7c4613d495f31f016f296307f6: Status 404 returned error can't find the container with id d6c24d65eaf37cb34a05a91dd7cbdabc71a82f7c4613d495f31f016f296307f6
Sep 17 23:10:40.446127 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.446098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x9277"
Sep 17 23:10:40.455087 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.455055    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5aa21c_1af6_47e6_b02e_1417aecdc5ad.slice/crio-021161948dd610e493feab8a54695701bd5a655f7e140340cb2a02f6ee7586ac WatchSource:0}: Error finding container 021161948dd610e493feab8a54695701bd5a655f7e140340cb2a02f6ee7586ac: Status 404 returned error can't find the container with id 021161948dd610e493feab8a54695701bd5a655f7e140340cb2a02f6ee7586ac
Sep 17 23:10:40.474042 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.473999    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9fctl"
Sep 17 23:10:40.479102 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.479081    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:10:40.484303 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.484278    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f1f9aa_a2cd_456e_9606_fdb3218caede.slice/crio-cb8ee96535179d1ab125a78d884f2fe9d3fe1897e45628260a9c686d53bb9374 WatchSource:0}: Error finding container cb8ee96535179d1ab125a78d884f2fe9d3fe1897e45628260a9c686d53bb9374: Status 404 returned error can't find the container with id cb8ee96535179d1ab125a78d884f2fe9d3fe1897e45628260a9c686d53bb9374
Sep 17 23:10:40.488956 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:10:40.488921    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee7a800_e764_412e_9826_9d242b39b039.slice/crio-6f49b7ae1f1148d59a54516f7c07a2e917d0d6cd928f827960d907d9d79589be WatchSource:0}: Error finding container 6f49b7ae1f1148d59a54516f7c07a2e917d0d6cd928f827960d907d9d79589be: Status 404 returned error can't find the container with id 6f49b7ae1f1148d59a54516f7c07a2e917d0d6cd928f827960d907d9d79589be
Sep 17 23:10:40.653297 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.653097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:40.653297 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.653172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:40.653297 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.653280    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:40.653574 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.653290    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:40.653574 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.653368    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs podName:c73226c0-ffce-43af-89c5-e7197f34364e nodeName:}" failed. No retries permitted until 2026-09-17 23:10:41.653348048 +0000 UTC m=+4.587793782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs") pod "network-metrics-daemon-pk7p2" (UID: "c73226c0-ffce-43af-89c5-e7197f34364e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:40.653574 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.653386    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret podName:986816f6-1fb8-429d-a4aa-c4778d601b56 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:41.6533749 +0000 UTC m=+4.587820642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret") pod "global-pull-secret-syncer-pzttd" (UID: "986816f6-1fb8-429d-a4aa-c4778d601b56") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:40.755228 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.754565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:40.755228 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.754752    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:40.755228 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.754773    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:40.755228 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.754787    2583 projected.go:196] Error preparing data for projected volume kube-api-access-cpr47 for pod openshift-network-diagnostics/network-check-target-b224x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:40.755228 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:40.754852    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47 podName:be180008-7f40-4e3c-b31a-bba2323922ef nodeName:}" failed. No retries permitted until 2026-09-17 23:10:41.754830217 +0000 UTC m=+4.689275960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpr47" (UniqueName: "kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47") pod "network-check-target-b224x" (UID: "be180008-7f40-4e3c-b31a-bba2323922ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:40.796165 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:40.796132    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:41.152284 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.152229    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:41.152284 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.152280    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:41.153069 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.152373    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:41.153069 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.152474    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:41.162282 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.162131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xcwf9" event={"ID":"b4e89f46-8753-4be5-be61-9ccbaf154c6f","Type":"ContainerStarted","Data":"25b360840f6fa4c4e8a9f26772f8a2ea8bb4ff8eee0f946fba07ae7e4d21122d"}
Sep 17 23:10:41.165342 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.165109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9fctl" event={"ID":"d9f1f9aa-a2cd-456e-9606-fdb3218caede","Type":"ContainerStarted","Data":"cb8ee96535179d1ab125a78d884f2fe9d3fe1897e45628260a9c686d53bb9374"}
Sep 17 23:10:41.168500 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.168408    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:40 +0000 UTC" deadline="2028-05-27 12:50:46.280042016 +0000 UTC"
Sep 17 23:10:41.168500 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.168445    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14821h40m5.11160415s"
Sep 17 23:10:41.176228 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.176130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dlnfn" event={"ID":"1aced041-ad96-4073-8c9a-dc28c37ea129","Type":"ContainerStarted","Data":"d6c24d65eaf37cb34a05a91dd7cbdabc71a82f7c4613d495f31f016f296307f6"}
Sep 17 23:10:41.180967 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.180923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7wwwj" event={"ID":"79e79ce8-102c-4719-9637-12f50a0382c3","Type":"ContainerStarted","Data":"ec97545deaed298d5ad337065f8355065b3e849f45d8a403ddff5d32696206e2"}
Sep 17 23:10:41.184428 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.184381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr" event={"ID":"a3e1aec2-2b62-46a2-8c82-93456e200ba8","Type":"ContainerStarted","Data":"d689df9a7d945733873ec4b88577006bb61b5792085dc63bf88dc576db314b0f"}
Sep 17 23:10:41.191960 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.191921    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal" event={"ID":"27ea57f9def2c8b7b50c3726e00d3bc7","Type":"ContainerStarted","Data":"851b5ecd3edff7234b1575c3328ba40a97927f4f184d1de8ff54c14734c04931"}
Sep 17 23:10:41.194921 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.194886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal" event={"ID":"c587c9980c41ab67e0f45c632018ea28","Type":"ContainerStarted","Data":"1b9588ee7534b789e1dfad03cb35f6055d784cc336eed76d650244dc6e12c8c8"}
Sep 17 23:10:41.203868 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.203753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerStarted","Data":"6f49b7ae1f1148d59a54516f7c07a2e917d0d6cd928f827960d907d9d79589be"}
Sep 17 23:10:41.206276 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.206168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x9277" event={"ID":"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad","Type":"ContainerStarted","Data":"021161948dd610e493feab8a54695701bd5a655f7e140340cb2a02f6ee7586ac"}
Sep 17 23:10:41.214037 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.213990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hfkwd" event={"ID":"d2c12fff-e88e-47a7-95d5-3897d1c5133a","Type":"ContainerStarted","Data":"2a9889921cddef325410b914595ebc774dd53938a3d3763011a5523f84ef79f8"}
Sep 17 23:10:41.225424 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.225384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnrdw" event={"ID":"99314a8b-f4ea-4521-ad1f-6d152fd397fb","Type":"ContainerStarted","Data":"174856722d0d7e1a634061760c2d9e986078f99089f26f1cac6dc8c539b3b2f3"}
Sep 17 23:10:41.663712 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.663670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:41.663906 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.663752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:41.663906 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.663899    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:41.664003 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.663962    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs podName:c73226c0-ffce-43af-89c5-e7197f34364e nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.663945026 +0000 UTC m=+6.598390778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs") pod "network-metrics-daemon-pk7p2" (UID: "c73226c0-ffce-43af-89c5-e7197f34364e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:41.664398 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.664376    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:41.664491 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.664428    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret podName:986816f6-1fb8-429d-a4aa-c4778d601b56 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.664412652 +0000 UTC m=+6.598858394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret") pod "global-pull-secret-syncer-pzttd" (UID: "986816f6-1fb8-429d-a4aa-c4778d601b56") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:41.764658 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:41.764618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:41.764883 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.764865    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:41.764954 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.764896    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:41.764954 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.764932    2583 projected.go:196] Error preparing data for projected volume kube-api-access-cpr47 for pod openshift-network-diagnostics/network-check-target-b224x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:41.765068 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:41.764995    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47 podName:be180008-7f40-4e3c-b31a-bba2323922ef nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.764976292 +0000 UTC m=+6.699422035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpr47" (UniqueName: "kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47") pod "network-check-target-b224x" (UID: "be180008-7f40-4e3c-b31a-bba2323922ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:42.151731 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:42.151700    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:42.151907 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:42.151836    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:42.168703 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:42.168655    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:40 +0000 UTC" deadline="2028-04-11 03:16:52.469527019 +0000 UTC"
Sep 17 23:10:42.168703 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:42.168702    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13708h6m10.300830537s"
Sep 17 23:10:42.517938 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:42.517855    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:43.154122 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:43.154078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:43.154327 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.154226    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:43.154705 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:43.154667    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:43.154961 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.154782    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:43.682645 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:43.682570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:43.682645 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:43.682646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:43.683248 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.682773    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:43.683248 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.682842    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs podName:c73226c0-ffce-43af-89c5-e7197f34364e nodeName:}" failed. No retries permitted until 2026-09-17 23:10:47.682822399 +0000 UTC m=+10.617268147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs") pod "network-metrics-daemon-pk7p2" (UID: "c73226c0-ffce-43af-89c5-e7197f34364e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:43.683248 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.683139    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:43.683248 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.683201    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret podName:986816f6-1fb8-429d-a4aa-c4778d601b56 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:47.683172904 +0000 UTC m=+10.617618632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret") pod "global-pull-secret-syncer-pzttd" (UID: "986816f6-1fb8-429d-a4aa-c4778d601b56") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:43.784053 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:43.783987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:43.784275 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.784192    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:43.784275 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.784217    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:43.784275 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.784230    2583 projected.go:196] Error preparing data for projected volume kube-api-access-cpr47 for pod openshift-network-diagnostics/network-check-target-b224x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:43.784480 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:43.784293    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47 podName:be180008-7f40-4e3c-b31a-bba2323922ef nodeName:}" failed. No retries permitted until 2026-09-17 23:10:47.784275965 +0000 UTC m=+10.718721701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpr47" (UniqueName: "kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47") pod "network-check-target-b224x" (UID: "be180008-7f40-4e3c-b31a-bba2323922ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:44.151625 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:44.151578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:44.151826 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:44.151737    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:45.151607 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:45.151561    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:45.152103 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:45.151697    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:45.152103 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:45.151756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:45.152103 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:45.151890    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:46.152211 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:46.152172    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:46.152690 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:46.152328    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:47.156151 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:47.155654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:47.156151 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:47.155703    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:47.156151 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.155788    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:47.156151 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.155906    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:47.719943 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:47.719904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:47.720167 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:47.719983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:47.720167 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.720119    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:47.720273 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.720175    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:47.720273 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.720208    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs podName:c73226c0-ffce-43af-89c5-e7197f34364e nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.720184671 +0000 UTC m=+18.654630400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs") pod "network-metrics-daemon-pk7p2" (UID: "c73226c0-ffce-43af-89c5-e7197f34364e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:47.720385 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.720308    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret podName:986816f6-1fb8-429d-a4aa-c4778d601b56 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.720273847 +0000 UTC m=+18.654719576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret") pod "global-pull-secret-syncer-pzttd" (UID: "986816f6-1fb8-429d-a4aa-c4778d601b56") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:47.821228 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:47.821183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:47.821422 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.821400    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:47.821466 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.821432    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:47.821466 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.821442    2583 projected.go:196] Error preparing data for projected volume kube-api-access-cpr47 for pod openshift-network-diagnostics/network-check-target-b224x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:47.821541 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:47.821493    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47 podName:be180008-7f40-4e3c-b31a-bba2323922ef nodeName:}" failed. No retries permitted until 2026-09-17 23:10:55.82147922 +0000 UTC m=+18.755924948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpr47" (UniqueName: "kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47") pod "network-check-target-b224x" (UID: "be180008-7f40-4e3c-b31a-bba2323922ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:48.152128 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:48.152089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:48.152330 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:48.152226    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:49.153005 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:49.152890    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:49.153525 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:49.153046    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:49.153525 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:49.153108    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:49.153525 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:49.153190    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:50.152058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:50.151999    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:50.152266 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:50.152187    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:51.151954 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:51.151911    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:51.152448 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:51.152066    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:51.152448 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:51.152108    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:51.152448 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:51.152232    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:52.151736 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:52.151697    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:52.151947 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:52.151848    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:53.156930 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:53.156891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:53.156930 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:53.156923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:53.157577 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:53.157016    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:53.157577 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:53.157140    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:54.151398 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:54.151364    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:54.151559 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:54.151486    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:55.155712 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:55.155629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:55.156164 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.155739    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:55.156164 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:55.155756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:55.156164 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.155879    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:55.783054 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:55.782938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:55.783054 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:55.783033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:55.783417 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.783112    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:55.783417 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.783139    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:55.783417 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.783192    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret podName:986816f6-1fb8-429d-a4aa-c4778d601b56 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:11.783171395 +0000 UTC m=+34.717617137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret") pod "global-pull-secret-syncer-pzttd" (UID: "986816f6-1fb8-429d-a4aa-c4778d601b56") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:55.783417 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.783212    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs podName:c73226c0-ffce-43af-89c5-e7197f34364e nodeName:}" failed. No retries permitted until 2026-09-17 23:11:11.783201969 +0000 UTC m=+34.717647697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs") pod "network-metrics-daemon-pk7p2" (UID: "c73226c0-ffce-43af-89c5-e7197f34364e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:55.884135 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:55.884095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:55.884315 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.884281    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:55.884315 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.884308    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:55.884403 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.884324    2583 projected.go:196] Error preparing data for projected volume kube-api-access-cpr47 for pod openshift-network-diagnostics/network-check-target-b224x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:55.884403 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:55.884392    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47 podName:be180008-7f40-4e3c-b31a-bba2323922ef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:11.884372754 +0000 UTC m=+34.818818487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpr47" (UniqueName: "kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47") pod "network-check-target-b224x" (UID: "be180008-7f40-4e3c-b31a-bba2323922ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:56.151574 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:56.151473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:56.151743 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:56.151638    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:57.151440 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:57.151395    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:57.151440 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:57.151438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:57.152002 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:57.151526    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:57.152002 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:57.151663    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:10:58.151830 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:58.151782    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:10:58.152316 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:58.151915    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:10:59.151781 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:59.151745    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:10:59.151960 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:10:59.151834    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:10:59.151960 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:59.151857    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:10:59.152379 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:10:59.151953    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:11:00.151835 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.151615    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:00.152014 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:00.151934    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:11:00.278521 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.278483    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal" event={"ID":"c587c9980c41ab67e0f45c632018ea28","Type":"ContainerStarted","Data":"fd6eace156185b0f94c2c98cd4482554ba77c3ce552a9c10379507848ab91547"}
Sep 17 23:11:00.281348 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.281322    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:11:00.281759 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.281736    2583 generic.go:356] "Generic (PLEG): container finished" podID="6ee7a800-e764-412e-9826-9d242b39b039" containerID="5915dbe918954419f62f42bf5de5c8e2cb1991e34586e22624c88c897e61b577" exitCode=1
Sep 17 23:11:00.281845 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.281815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerStarted","Data":"2e2177801a472707017fe7b358c4c057d391e3fe2e892ff26a48352f9de174f6"}
Sep 17 23:11:00.281930 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.281913    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerStarted","Data":"ba5517b65fb082308010a2b91119a07b7394c0fdd8f64b6714fe68f2b13d0c79"}
Sep 17 23:11:00.281995 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.281973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerStarted","Data":"05c4c6ca5ed024728a65971c635cbb74cd8a7d71b0c9678d24d9ba541a57ea3c"}
Sep 17 23:11:00.282071 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.282046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerStarted","Data":"1fb681e90d7e4e9598dc73275577761042eb1dc4154df87ce4826de3a983c448"}
Sep 17 23:11:00.282129 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.282107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerDied","Data":"5915dbe918954419f62f42bf5de5c8e2cb1991e34586e22624c88c897e61b577"}
Sep 17 23:11:00.282179 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.282162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerStarted","Data":"3b93b1d56cd04cd094490687661e81830cfe04875f891cdb2dd32f629f1e0f9f"}
Sep 17 23:11:00.283365 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.283340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hfkwd" event={"ID":"d2c12fff-e88e-47a7-95d5-3897d1c5133a","Type":"ContainerStarted","Data":"81f0cec10b39ff9665752d7df809d1a1fcfec6f9cae7b475303cb107426536c4"}
Sep 17 23:11:00.284758 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.284736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7wwwj" event={"ID":"79e79ce8-102c-4719-9637-12f50a0382c3","Type":"ContainerStarted","Data":"0992f6d31e0212b07889b57263d09d4d696995e79a692884dff0ea134605e7ce"}
Sep 17 23:11:00.318127 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.318059    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-96.ec2.internal" podStartSLOduration=21.31804296 podStartE2EDuration="21.31804296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:00.297332782 +0000 UTC m=+23.231778533" watchObservedRunningTime="2026-09-17 23:11:00.31804296 +0000 UTC m=+23.252488706"
Sep 17 23:11:00.337571 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.337510    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7wwwj" podStartSLOduration=2.4863305860000002 podStartE2EDuration="21.337487583s" totalImagesPullingTime="18.851156997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:00.318251824 +0000 UTC m=+23.252697585" watchObservedRunningTime="2026-09-17 23:11:00.337487583 +0000 UTC m=+23.271933325"
Sep 17 23:11:00.337766 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:00.337744    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hfkwd" podStartSLOduration=2.543645564 podStartE2EDuration="21.337737761s" totalImagesPullingTime="18.794092197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:00.337632823 +0000 UTC m=+23.272078555" watchObservedRunningTime="2026-09-17 23:11:00.337737761 +0000 UTC m=+23.272183510"
Sep 17 23:11:01.152131 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.152090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:01.152694 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.152093    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:01.152694 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:01.152202    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:11:01.152694 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:01.152287    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:11:01.288212 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.288164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x9277" event={"ID":"fb5aa21c-1af6-47e6-b02e-1417aecdc5ad","Type":"ContainerStarted","Data":"97ff78af11cdd6c32ed22d2640dbeca27906149665502493deb27f25228d8fb4"}
Sep 17 23:11:01.289806 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.289779    2583 generic.go:356] "Generic (PLEG): container finished" podID="99314a8b-f4ea-4521-ad1f-6d152fd397fb" containerID="2f20c5bdc9d470381d0d96a0fee0a53d53276a093790c42e4cc8bcaf5823a271" exitCode=0
Sep 17 23:11:01.289954 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.289854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnrdw" event={"ID":"99314a8b-f4ea-4521-ad1f-6d152fd397fb","Type":"ContainerDied","Data":"2f20c5bdc9d470381d0d96a0fee0a53d53276a093790c42e4cc8bcaf5823a271"}
Sep 17 23:11:01.292076 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.292004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xcwf9" event={"ID":"b4e89f46-8753-4be5-be61-9ccbaf154c6f","Type":"ContainerStarted","Data":"5765323a4d2d77adcaa1c59c95c3ed332a200763f67b0822f259202687a6c0a9"}
Sep 17 23:11:01.293493 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.293467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9fctl" event={"ID":"d9f1f9aa-a2cd-456e-9606-fdb3218caede","Type":"ContainerStarted","Data":"20a16dc07e9c9209cdbc56c46bc78f2761b5c571dc0d42feac2de71a49bda1c1"}
Sep 17 23:11:01.294935 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.294909    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dlnfn" event={"ID":"1aced041-ad96-4073-8c9a-dc28c37ea129","Type":"ContainerStarted","Data":"6f159c506693742b1f21c01d7859db8165dce5dc1f157eb99e454f1a247d7cec"}
Sep 17 23:11:01.296449 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.296423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr" event={"ID":"a3e1aec2-2b62-46a2-8c82-93456e200ba8","Type":"ContainerStarted","Data":"0c68da306d63ac0071a4b585e105968babefee7595f1ba91a92213783a1f76f9"}
Sep 17 23:11:01.298049 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.298005    2583 generic.go:356] "Generic (PLEG): container finished" podID="27ea57f9def2c8b7b50c3726e00d3bc7" containerID="8c593be3f7ebe0973b1be8cc4a4ab4cd54effff75e5ad5562d49fe9624fc13b3" exitCode=0
Sep 17 23:11:01.298148 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.298102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal" event={"ID":"27ea57f9def2c8b7b50c3726e00d3bc7","Type":"ContainerDied","Data":"8c593be3f7ebe0973b1be8cc4a4ab4cd54effff75e5ad5562d49fe9624fc13b3"}
Sep 17 23:11:01.345178 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.345113    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x9277" podStartSLOduration=3.61630013 podStartE2EDuration="22.345090325s" totalImagesPullingTime="18.728790195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.316861849 +0000 UTC m=+24.251307596" watchObservedRunningTime="2026-09-17 23:11:01.345090325 +0000 UTC m=+24.279536076"
Sep 17 23:11:01.345364 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.345278    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9fctl" podStartSLOduration=3.645522037 podStartE2EDuration="22.34527019s" totalImagesPullingTime="18.699748153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.344670612 +0000 UTC m=+24.279116363" watchObservedRunningTime="2026-09-17 23:11:01.34527019 +0000 UTC m=+24.279715941"
Sep 17 23:11:01.438418 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.438070    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dlnfn" podStartSLOduration=3.660286533 podStartE2EDuration="22.43804642s" totalImagesPullingTime="18.777759887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.435558864 +0000 UTC m=+24.370004669" watchObservedRunningTime="2026-09-17 23:11:01.43804642 +0000 UTC m=+24.372492171"
Sep 17 23:11:01.456928 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.456865    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xcwf9" podStartSLOduration=3.613952334 podStartE2EDuration="22.456843776s" totalImagesPullingTime="18.842891442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.456305469 +0000 UTC m=+24.390751220" watchObservedRunningTime="2026-09-17 23:11:01.456843776 +0000 UTC m=+24.391289521"
Sep 17 23:11:01.965185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:01.965149    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:11:02.107059 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:02.106865    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:11:01.965170743Z","UUID":"78685fd0-1922-4f03-8bc6-aaab3995cc86","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:11:02.108837 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:02.108794    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:11:02.109012 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:02.108846    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:11:02.151389 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:02.151353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:02.151571 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:02.151474    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:11:02.303196 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:02.303155    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr" event={"ID":"a3e1aec2-2b62-46a2-8c82-93456e200ba8","Type":"ContainerStarted","Data":"c518128ee1056bba8949bdaeb06925a27af80e90469d174de89601d3685f56e5"}
Sep 17 23:11:02.305314 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:02.305276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal" event={"ID":"27ea57f9def2c8b7b50c3726e00d3bc7","Type":"ContainerStarted","Data":"e82d9f082c6ebcac95bb23046a6e153dbbccd9cf81c03b6f66925fcf5c32a9cd"}
Sep 17 23:11:03.079268 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:03.079226    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:11:03.079910 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:03.079887    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:11:03.098434 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:03.098376    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-96.ec2.internal" podStartSLOduration=24.098358522 podStartE2EDuration="24.098358522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:02.324710984 +0000 UTC m=+25.259156736" watchObservedRunningTime="2026-09-17 23:11:03.098358522 +0000 UTC m=+26.032804272"
Sep 17 23:11:03.152151 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:03.152121    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:03.152324 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:03.152158    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:03.152324 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:03.152236    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:11:03.152437 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:03.152370    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:11:03.310010 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:03.309969    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr" event={"ID":"a3e1aec2-2b62-46a2-8c82-93456e200ba8","Type":"ContainerStarted","Data":"f6f340a3121e2075b58c75f61e823a3436d63fa0e1c2c89dda2b6462584dc9eb"}
Sep 17 23:11:03.313741 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:03.313716    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:11:03.314217 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:03.314184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerStarted","Data":"c8f5302fe8511937eac9eae370087f9668b02c070445152522326ef216143aa1"}
Sep 17 23:11:03.331405 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:03.331343    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4ftr" podStartSLOduration=3.38670182 podStartE2EDuration="24.331319845s" totalImagesPullingTime="20.944618025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:03.330562224 +0000 UTC m=+26.265007965" watchObservedRunningTime="2026-09-17 23:11:03.331319845 +0000 UTC m=+26.265765598"
Sep 17 23:11:04.151181 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:04.151140    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:04.151394 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:04.151297    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:11:04.316739 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:04.316706    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 23:11:05.151308 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:05.151272    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:05.151483 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:05.151272    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:05.151483 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:05.151412    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:11:05.151613 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:05.151489    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:11:06.151940 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:06.151756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:06.152475 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:06.152045    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:11:06.324750 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:06.324722    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:11:06.325168 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:06.325135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerStarted","Data":"f4fcd2ef72c209ee53b08d636879a92bbaca79b9e2b604b9724aecb5ea0df422"}
Sep 17 23:11:06.325515 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:06.325486    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:11:06.325679 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:06.325657    2583 scope.go:117] "RemoveContainer" containerID="5915dbe918954419f62f42bf5de5c8e2cb1991e34586e22624c88c897e61b577"
Sep 17 23:11:06.327310 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:06.327284    2583 generic.go:356] "Generic (PLEG): container finished" podID="99314a8b-f4ea-4521-ad1f-6d152fd397fb" containerID="0ddc5882e73e4dbb53b4a3aaff600753309e02beb2c35497b7884a9d6cb84053" exitCode=0
Sep 17 23:11:06.327429 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:06.327338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnrdw" event={"ID":"99314a8b-f4ea-4521-ad1f-6d152fd397fb","Type":"ContainerDied","Data":"0ddc5882e73e4dbb53b4a3aaff600753309e02beb2c35497b7884a9d6cb84053"}
Sep 17 23:11:06.343293 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:06.343261    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:11:07.151811 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.151768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:07.152009 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.151816    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:07.152009 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:07.151900    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:11:07.152613 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:07.152043    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:11:07.334522 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.334455    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:11:07.338308 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.338265    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" event={"ID":"6ee7a800-e764-412e-9826-9d242b39b039","Type":"ContainerStarted","Data":"c2ae0eb4d79abd41faac3ca13011ee40a800f40372c8bf72b19b5cfbd3263cd6"}
Sep 17 23:11:07.338573 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.338556    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:11:07.338650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.338626    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:11:07.341065 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.341017    2583 generic.go:356] "Generic (PLEG): container finished" podID="99314a8b-f4ea-4521-ad1f-6d152fd397fb" containerID="c55d6040ebb5b9626a02402d5babfe4684ae7bc8db73c234ced664f350fd6a72" exitCode=0
Sep 17 23:11:07.341227 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.341083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnrdw" event={"ID":"99314a8b-f4ea-4521-ad1f-6d152fd397fb","Type":"ContainerDied","Data":"c55d6040ebb5b9626a02402d5babfe4684ae7bc8db73c234ced664f350fd6a72"}
Sep 17 23:11:07.355201 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.355171    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:11:07.436178 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.436120    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h" podStartSLOduration=9.710478079 podStartE2EDuration="28.436104801s" totalImagesPullingTime="18.725626722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:07.401091442 +0000 UTC m=+30.335537193" watchObservedRunningTime="2026-09-17 23:11:07.436104801 +0000 UTC m=+30.370550555"
Sep 17 23:11:07.637407 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.637321    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b224x"]
Sep 17 23:11:07.637566 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.637442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:07.637566 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:07.637531    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:11:07.637947 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.637921    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pzttd"]
Sep 17 23:11:07.638072 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.638049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:07.638169 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:07.638145    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:11:07.642490 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.642460    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pk7p2"]
Sep 17 23:11:07.642613 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:07.642587    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:07.642689 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:07.642673    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:11:08.345945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:08.345909    2583 generic.go:356] "Generic (PLEG): container finished" podID="99314a8b-f4ea-4521-ad1f-6d152fd397fb" containerID="599dfc9a3addb39a84de71eef804689058939dc2e19adf37d519d8bc648eb63e" exitCode=0
Sep 17 23:11:08.346338 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:08.345995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnrdw" event={"ID":"99314a8b-f4ea-4521-ad1f-6d152fd397fb","Type":"ContainerDied","Data":"599dfc9a3addb39a84de71eef804689058939dc2e19adf37d519d8bc648eb63e"}
Sep 17 23:11:09.153219 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:09.153179    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:09.153403 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:09.153289    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:11:09.153467 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:09.153419    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:09.153550 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:09.153525    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:11:09.153602 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:09.153585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:09.153720 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:09.153700    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:11:09.392934 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:09.392899    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:11:09.393482 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:09.393068    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 23:11:09.393675 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:09.393658    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xcwf9"
Sep 17 23:11:11.155702 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:11.155662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:11.156165 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:11.155798    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:11.156165 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.155817    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pk7p2" podUID="c73226c0-ffce-43af-89c5-e7197f34364e"
Sep 17 23:11:11.156165 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:11.155835    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:11.156165 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.155920    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b224x" podUID="be180008-7f40-4e3c-b31a-bba2323922ef"
Sep 17 23:11:11.156165 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.155996    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzttd" podUID="986816f6-1fb8-429d-a4aa-c4778d601b56"
Sep 17 23:11:11.803387 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:11.803343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:11.803584 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:11.803413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:11.803584 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.803501    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:11.803584 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.803571    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs podName:c73226c0-ffce-43af-89c5-e7197f34364e nodeName:}" failed. No retries permitted until 2026-09-17 23:11:43.803554339 +0000 UTC m=+66.738000080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs") pod "network-metrics-daemon-pk7p2" (UID: "c73226c0-ffce-43af-89c5-e7197f34364e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:11.803754 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.803501    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:11.803754 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.803656    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret podName:986816f6-1fb8-429d-a4aa-c4778d601b56 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:43.803642613 +0000 UTC m=+66.738088354 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret") pod "global-pull-secret-syncer-pzttd" (UID: "986816f6-1fb8-429d-a4aa-c4778d601b56") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:11.904190 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:11.904148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:11.904385 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.904336    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:11.904385 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.904362    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:11.904385 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.904374    2583 projected.go:196] Error preparing data for projected volume kube-api-access-cpr47 for pod openshift-network-diagnostics/network-check-target-b224x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:11.904541 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:11.904441    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47 podName:be180008-7f40-4e3c-b31a-bba2323922ef nodeName:}" failed. No retries permitted until 2026-09-17 23:11:43.904420508 +0000 UTC m=+66.838866256 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cpr47" (UniqueName: "kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47") pod "network-check-target-b224x" (UID: "be180008-7f40-4e3c-b31a-bba2323922ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:12.562676 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.562643    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-96.ec2.internal" event="NodeReady"
Sep 17 23:11:12.563169 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.562806    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:11:12.619039 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.618987    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74dcc75879-rt6s8"]
Sep 17 23:11:12.621847 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.621814    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.627325 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.627298    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 23:11:12.630014 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.629995    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 23:11:12.630760 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.630742    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6gj7c\""
Sep 17 23:11:12.631191 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.631175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 23:11:12.642178 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.642143    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 23:11:12.659996 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.659953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74dcc75879-rt6s8"]
Sep 17 23:11:12.660140 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.660039    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-548p9"]
Sep 17 23:11:12.663267 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.663218    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r5r48"]
Sep 17 23:11:12.663397 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.663333    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.666267 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.666248    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r5r48"
Sep 17 23:11:12.669771 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.668435    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:11:12.669771 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.668754    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xxcjw\""
Sep 17 23:11:12.669771 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.668982    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:11:12.670952 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.670929    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:11:12.671129 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.671080    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qctjw\""
Sep 17 23:11:12.671976 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.671860    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:11:12.672117 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.672101    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:11:12.688569 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.688096    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-548p9"]
Sep 17 23:11:12.688569 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.688453    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r5r48"]
Sep 17 23:11:12.710497 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.710469    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-bound-sa-token\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.710652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.710504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09eaea72-e75d-4676-8fc0-e528b568a81f-ca-trust-extracted\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.710652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.710521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-trusted-ca\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.710652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.710564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-image-registry-private-configuration\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.710810 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.710651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.710810 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.710693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-certificates\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.710810 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.710725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9n9\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-kube-api-access-nw9n9\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.710810 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.710758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-installation-pull-secrets\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.811898 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.811861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79530f25-f88f-4dfa-8077-90c9a35211dd-cert\") pod \"ingress-canary-r5r48\" (UID: \"79530f25-f88f-4dfa-8077-90c9a35211dd\") " pod="openshift-ingress-canary/ingress-canary-r5r48"
Sep 17 23:11:12.811898 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.811909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.812231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.811931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-certificates\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.812231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.811958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9n9\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-kube-api-access-nw9n9\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.812231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.811983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ss9\" (UniqueName: \"kubernetes.io/projected/79530f25-f88f-4dfa-8077-90c9a35211dd-kube-api-access-c9ss9\") pod \"ingress-canary-r5r48\" (UID: \"79530f25-f88f-4dfa-8077-90c9a35211dd\") " pod="openshift-ingress-canary/ingress-canary-r5r48"
Sep 17 23:11:12.812231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwdx\" (UniqueName: \"kubernetes.io/projected/f200acdd-f4e1-4afa-a742-4d5981e022cb-kube-api-access-pcwdx\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.812231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f200acdd-f4e1-4afa-a742-4d5981e022cb-tmp-dir\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.812231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-installation-pull-secrets\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.812231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-bound-sa-token\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.812611 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f200acdd-f4e1-4afa-a742-4d5981e022cb-config-volume\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.812611 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812277    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.812611 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09eaea72-e75d-4676-8fc0-e528b568a81f-ca-trust-extracted\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.812611 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-trusted-ca\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.812611 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-image-registry-private-configuration\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.812611 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:12.812366    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:12.812611 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:12.812403    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dcc75879-rt6s8: secret "image-registry-tls" not found
Sep 17 23:11:12.812611 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:12.812461    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls podName:09eaea72-e75d-4676-8fc0-e528b568a81f nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.312443471 +0000 UTC m=+36.246889199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls") pod "image-registry-74dcc75879-rt6s8" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f") : secret "image-registry-tls" not found
Sep 17 23:11:12.812611 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-certificates\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.813147 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.812684    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09eaea72-e75d-4676-8fc0-e528b568a81f-ca-trust-extracted\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.813512 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.813487    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-trusted-ca\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.816512 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.816380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-installation-pull-secrets\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.816623 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.816396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-image-registry-private-configuration\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.822053 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.822010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9n9\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-kube-api-access-nw9n9\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.825694 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.825658    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-bound-sa-token\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:12.912856 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.912816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79530f25-f88f-4dfa-8077-90c9a35211dd-cert\") pod \"ingress-canary-r5r48\" (UID: \"79530f25-f88f-4dfa-8077-90c9a35211dd\") " pod="openshift-ingress-canary/ingress-canary-r5r48"
Sep 17 23:11:12.913054 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.912868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ss9\" (UniqueName: \"kubernetes.io/projected/79530f25-f88f-4dfa-8077-90c9a35211dd-kube-api-access-c9ss9\") pod \"ingress-canary-r5r48\" (UID: \"79530f25-f88f-4dfa-8077-90c9a35211dd\") " pod="openshift-ingress-canary/ingress-canary-r5r48"
Sep 17 23:11:12.913054 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.912886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwdx\" (UniqueName: \"kubernetes.io/projected/f200acdd-f4e1-4afa-a742-4d5981e022cb-kube-api-access-pcwdx\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.913143 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.913047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f200acdd-f4e1-4afa-a742-4d5981e022cb-tmp-dir\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.913143 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.913130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f200acdd-f4e1-4afa-a742-4d5981e022cb-config-volume\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.913227 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.913154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.913273 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:12.913252    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:12.913331 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:12.913319    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls podName:f200acdd-f4e1-4afa-a742-4d5981e022cb nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.413300924 +0000 UTC m=+36.347746652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls") pod "dns-default-548p9" (UID: "f200acdd-f4e1-4afa-a742-4d5981e022cb") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:12.913402 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.913384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f200acdd-f4e1-4afa-a742-4d5981e022cb-tmp-dir\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.913705 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.913686    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f200acdd-f4e1-4afa-a742-4d5981e022cb-config-volume\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.915514 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.915495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79530f25-f88f-4dfa-8077-90c9a35211dd-cert\") pod \"ingress-canary-r5r48\" (UID: \"79530f25-f88f-4dfa-8077-90c9a35211dd\") " pod="openshift-ingress-canary/ingress-canary-r5r48"
Sep 17 23:11:12.928303 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.928277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ss9\" (UniqueName: \"kubernetes.io/projected/79530f25-f88f-4dfa-8077-90c9a35211dd-kube-api-access-c9ss9\") pod \"ingress-canary-r5r48\" (UID: \"79530f25-f88f-4dfa-8077-90c9a35211dd\") " pod="openshift-ingress-canary/ingress-canary-r5r48"
Sep 17 23:11:12.929178 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.929158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwdx\" (UniqueName: \"kubernetes.io/projected/f200acdd-f4e1-4afa-a742-4d5981e022cb-kube-api-access-pcwdx\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:12.989682 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:12.989646    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r5r48"
Sep 17 23:11:13.155610 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.155535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:13.155610 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.155563    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:13.155801 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.155641    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:13.159542 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.159517    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:13.159542 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.159540    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:13.162063 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.160289    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:13.162063 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.160828    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6c57\""
Sep 17 23:11:13.162063 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.161143    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:13.162063 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.161397    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wfwww\""
Sep 17 23:11:13.277394 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.277352    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r5r48"]
Sep 17 23:11:13.281501 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:11:13.281471    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79530f25_f88f_4dfa_8077_90c9a35211dd.slice/crio-54c3918428cc2683eea1badf4a3b489c3249d9d2016e4f12e916735d4f6d2cee WatchSource:0}: Error finding container 54c3918428cc2683eea1badf4a3b489c3249d9d2016e4f12e916735d4f6d2cee: Status 404 returned error can't find the container with id 54c3918428cc2683eea1badf4a3b489c3249d9d2016e4f12e916735d4f6d2cee
Sep 17 23:11:13.316483 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.316440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:13.316682 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:13.316602    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:13.316682 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:13.316621    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dcc75879-rt6s8: secret "image-registry-tls" not found
Sep 17 23:11:13.316682 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:13.316683    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls podName:09eaea72-e75d-4676-8fc0-e528b568a81f nodeName:}" failed. No retries permitted until 2026-09-17 23:11:14.316663777 +0000 UTC m=+37.251109516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls") pod "image-registry-74dcc75879-rt6s8" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f") : secret "image-registry-tls" not found
Sep 17 23:11:13.361753 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.361715    2583 generic.go:356] "Generic (PLEG): container finished" podID="99314a8b-f4ea-4521-ad1f-6d152fd397fb" containerID="8bc2f0bf3686397cfa170eca603ee5c3e12c7b54a911a74bcc93c3d3ea5471e7" exitCode=0
Sep 17 23:11:13.361917 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.361789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnrdw" event={"ID":"99314a8b-f4ea-4521-ad1f-6d152fd397fb","Type":"ContainerDied","Data":"8bc2f0bf3686397cfa170eca603ee5c3e12c7b54a911a74bcc93c3d3ea5471e7"}
Sep 17 23:11:13.363122 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.363091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r5r48" event={"ID":"79530f25-f88f-4dfa-8077-90c9a35211dd","Type":"ContainerStarted","Data":"54c3918428cc2683eea1badf4a3b489c3249d9d2016e4f12e916735d4f6d2cee"}
Sep 17 23:11:13.417334 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:13.417307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:13.417458 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:13.417437    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:13.417502 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:13.417492    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls podName:f200acdd-f4e1-4afa-a742-4d5981e022cb nodeName:}" failed. No retries permitted until 2026-09-17 23:11:14.417477515 +0000 UTC m=+37.351923243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls") pod "dns-default-548p9" (UID: "f200acdd-f4e1-4afa-a742-4d5981e022cb") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:14.325486 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:14.325442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:14.326352 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:14.325607    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:14.326352 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:14.325632    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dcc75879-rt6s8: secret "image-registry-tls" not found
Sep 17 23:11:14.326352 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:14.325691    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls podName:09eaea72-e75d-4676-8fc0-e528b568a81f nodeName:}" failed. No retries permitted until 2026-09-17 23:11:16.325676222 +0000 UTC m=+39.260121950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls") pod "image-registry-74dcc75879-rt6s8" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f") : secret "image-registry-tls" not found
Sep 17 23:11:14.370795 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:14.370753    2583 generic.go:356] "Generic (PLEG): container finished" podID="99314a8b-f4ea-4521-ad1f-6d152fd397fb" containerID="c615875ca7e3abdfacbea407f09fbe34c7ae2fe9bb1d1cfceb8081fb9918d22c" exitCode=0
Sep 17 23:11:14.370967 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:14.370833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnrdw" event={"ID":"99314a8b-f4ea-4521-ad1f-6d152fd397fb","Type":"ContainerDied","Data":"c615875ca7e3abdfacbea407f09fbe34c7ae2fe9bb1d1cfceb8081fb9918d22c"}
Sep 17 23:11:14.426739 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:14.426705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:14.426892 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:14.426852    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:14.426956 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:14.426914    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls podName:f200acdd-f4e1-4afa-a742-4d5981e022cb nodeName:}" failed. No retries permitted until 2026-09-17 23:11:16.426895281 +0000 UTC m=+39.361341233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls") pod "dns-default-548p9" (UID: "f200acdd-f4e1-4afa-a742-4d5981e022cb") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:15.377562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:15.377528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cnrdw" event={"ID":"99314a8b-f4ea-4521-ad1f-6d152fd397fb","Type":"ContainerStarted","Data":"9d4451ae4043ab4fd1b1cb74b08688b523375d9d7bf5f1d32066da496603c49f"}
Sep 17 23:11:15.415252 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:15.415188    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cnrdw" podStartSLOduration=7.783378577 podStartE2EDuration="36.415163847s" totalImagesPullingTime="28.63178527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:15.413636761 +0000 UTC m=+38.348082514" watchObservedRunningTime="2026-09-17 23:11:15.415163847 +0000 UTC m=+38.349609599"
Sep 17 23:11:16.341149 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:16.340907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:16.341328 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:16.341092    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:16.341328 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:16.341227    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dcc75879-rt6s8: secret "image-registry-tls" not found
Sep 17 23:11:16.341328 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:16.341293    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls podName:09eaea72-e75d-4676-8fc0-e528b568a81f nodeName:}" failed. No retries permitted until 2026-09-17 23:11:20.34127613 +0000 UTC m=+43.275721864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls") pod "image-registry-74dcc75879-rt6s8" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f") : secret "image-registry-tls" not found
Sep 17 23:11:16.381482 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:16.381441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r5r48" event={"ID":"79530f25-f88f-4dfa-8077-90c9a35211dd","Type":"ContainerStarted","Data":"b77d5b197c7c9f5b0368d1bda1d5c312f314d90b542cb9bc8b90a37b94fa7dcd"}
Sep 17 23:11:16.413629 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:16.413579    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r5r48" podStartSLOduration=1.965727783 podStartE2EDuration="4.41356196s" totalImagesPullingTime="2.447834177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:16.412798925 +0000 UTC m=+39.347244675" watchObservedRunningTime="2026-09-17 23:11:16.41356196 +0000 UTC m=+39.348007793"
Sep 17 23:11:16.442259 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:16.442224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:16.442428 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:16.442408    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:16.442490 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:16.442481    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls podName:f200acdd-f4e1-4afa-a742-4d5981e022cb nodeName:}" failed. No retries permitted until 2026-09-17 23:11:20.442462732 +0000 UTC m=+43.376908460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls") pod "dns-default-548p9" (UID: "f200acdd-f4e1-4afa-a742-4d5981e022cb") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:20.372908 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:20.372865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:20.373340 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:20.373009    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:20.373340 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:20.373041    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dcc75879-rt6s8: secret "image-registry-tls" not found
Sep 17 23:11:20.373340 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:20.373098    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls podName:09eaea72-e75d-4676-8fc0-e528b568a81f nodeName:}" failed. No retries permitted until 2026-09-17 23:11:28.373081715 +0000 UTC m=+51.307527447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls") pod "image-registry-74dcc75879-rt6s8" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f") : secret "image-registry-tls" not found
Sep 17 23:11:20.473796 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:20.473748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:20.473990 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:20.473900    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:20.473990 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:20.473964    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls podName:f200acdd-f4e1-4afa-a742-4d5981e022cb nodeName:}" failed. No retries permitted until 2026-09-17 23:11:28.473948203 +0000 UTC m=+51.408393931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls") pod "dns-default-548p9" (UID: "f200acdd-f4e1-4afa-a742-4d5981e022cb") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:28.431342 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:28.431298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:28.431774 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:28.431422    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:28.431774 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:28.431433    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dcc75879-rt6s8: secret "image-registry-tls" not found
Sep 17 23:11:28.431774 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:28.431485    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls podName:09eaea72-e75d-4676-8fc0-e528b568a81f nodeName:}" failed. No retries permitted until 2026-09-17 23:11:44.431469208 +0000 UTC m=+67.365914949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls") pod "image-registry-74dcc75879-rt6s8" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f") : secret "image-registry-tls" not found
Sep 17 23:11:28.532608 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:28.532579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:28.532737 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:28.532717    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:28.532789 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:28.532778    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls podName:f200acdd-f4e1-4afa-a742-4d5981e022cb nodeName:}" failed. No retries permitted until 2026-09-17 23:11:44.532762716 +0000 UTC m=+67.467208444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls") pod "dns-default-548p9" (UID: "f200acdd-f4e1-4afa-a742-4d5981e022cb") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:39.359752 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:39.359718    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp97h"
Sep 17 23:11:43.842064 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:43.842003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:43.842564 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:43.842097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:11:43.852072 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:43.852053    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:43.852233 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:43.852216    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:11:43.852293 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:43.852283    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs podName:c73226c0-ffce-43af-89c5-e7197f34364e nodeName:}" failed. No retries permitted until 2026-09-17 23:12:47.852267912 +0000 UTC m=+130.786713652 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs") pod "network-metrics-daemon-pk7p2" (UID: "c73226c0-ffce-43af-89c5-e7197f34364e") : secret "metrics-daemon-secret" not found
Sep 17 23:11:43.852619 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:43.852603    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:43.865596 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:43.865574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/986816f6-1fb8-429d-a4aa-c4778d601b56-original-pull-secret\") pod \"global-pull-secret-syncer-pzttd\" (UID: \"986816f6-1fb8-429d-a4aa-c4778d601b56\") " pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:43.943002 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:43.942975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:43.946809 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:43.946793    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:43.956367 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:43.956348    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:43.966371 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:43.966355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpr47\" (UniqueName: \"kubernetes.io/projected/be180008-7f40-4e3c-b31a-bba2323922ef-kube-api-access-cpr47\") pod \"network-check-target-b224x\" (UID: \"be180008-7f40-4e3c-b31a-bba2323922ef\") " pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:44.068776 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:44.068736    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzttd"
Sep 17 23:11:44.082818 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:44.082794    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wfwww\""
Sep 17 23:11:44.090754 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:44.090736    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:44.364198 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:44.364130    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pzttd"]
Sep 17 23:11:44.367264 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:11:44.367237    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986816f6_1fb8_429d_a4aa_c4778d601b56.slice/crio-e674244908f6d99faecfafb06b0962813b872e0071fd4f694222e50562b4294c WatchSource:0}: Error finding container e674244908f6d99faecfafb06b0962813b872e0071fd4f694222e50562b4294c: Status 404 returned error can't find the container with id e674244908f6d99faecfafb06b0962813b872e0071fd4f694222e50562b4294c
Sep 17 23:11:44.375469 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:44.375447    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b224x"]
Sep 17 23:11:44.378552 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:11:44.378524    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe180008_7f40_4e3c_b31a_bba2323922ef.slice/crio-b2c637cf30ce21ab0767d3eab34fed9bda8df24a1bf44ac9905dcb0c118950e8 WatchSource:0}: Error finding container b2c637cf30ce21ab0767d3eab34fed9bda8df24a1bf44ac9905dcb0c118950e8: Status 404 returned error can't find the container with id b2c637cf30ce21ab0767d3eab34fed9bda8df24a1bf44ac9905dcb0c118950e8
Sep 17 23:11:44.448353 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:44.448320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:11:44.448507 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:44.448474    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:44.448507 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:44.448489    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74dcc75879-rt6s8: secret "image-registry-tls" not found
Sep 17 23:11:44.448581 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:44.448557    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls podName:09eaea72-e75d-4676-8fc0-e528b568a81f nodeName:}" failed. No retries permitted until 2026-09-17 23:12:16.448536549 +0000 UTC m=+99.382982278 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls") pod "image-registry-74dcc75879-rt6s8" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f") : secret "image-registry-tls" not found
Sep 17 23:11:44.449989 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:44.449956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b224x" event={"ID":"be180008-7f40-4e3c-b31a-bba2323922ef","Type":"ContainerStarted","Data":"b2c637cf30ce21ab0767d3eab34fed9bda8df24a1bf44ac9905dcb0c118950e8"}
Sep 17 23:11:44.451002 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:44.450981    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pzttd" event={"ID":"986816f6-1fb8-429d-a4aa-c4778d601b56","Type":"ContainerStarted","Data":"e674244908f6d99faecfafb06b0962813b872e0071fd4f694222e50562b4294c"}
Sep 17 23:11:44.548663 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:44.548625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:11:44.548831 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:44.548788    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:44.548893 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:44.548882    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls podName:f200acdd-f4e1-4afa-a742-4d5981e022cb nodeName:}" failed. No retries permitted until 2026-09-17 23:12:16.548862429 +0000 UTC m=+99.483308157 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls") pod "dns-default-548p9" (UID: "f200acdd-f4e1-4afa-a742-4d5981e022cb") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:49.466982 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.466943    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b224x" event={"ID":"be180008-7f40-4e3c-b31a-bba2323922ef","Type":"ContainerStarted","Data":"bea19d7f633c158e4363a09ea6de450b18fb26e41095e91c5e1a9193e10ef4a7"}
Sep 17 23:11:49.467461 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.467128    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:11:49.469365 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.469335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pzttd" event={"ID":"986816f6-1fb8-429d-a4aa-c4778d601b56","Type":"ContainerStarted","Data":"50eb24965cf10f49be1b2232fb3c0dc1c005eb6840a57b8b6fd9f88f84a73e78"}
Sep 17 23:11:49.489125 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.489066    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b224x" podStartSLOduration=65.505898993 podStartE2EDuration="1m10.489048955s" totalImagesPullingTime="4.983149962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:49.488362442 +0000 UTC m=+72.422808197" watchObservedRunningTime="2026-09-17 23:11:49.489048955 +0000 UTC m=+72.423494704"
Sep 17 23:11:49.517529 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.517471    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pzttd" podStartSLOduration=65.518591627 podStartE2EDuration="1m10.51745554s" totalImagesPullingTime="4.998863913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:49.517127553 +0000 UTC m=+72.451573320" watchObservedRunningTime="2026-09-17 23:11:49.51745554 +0000 UTC m=+72.451901316"
Sep 17 23:11:49.879056 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.879005    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5d6966fbcb-qhw8d"]
Sep 17 23:11:49.882000 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.881975    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:49.890051 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:49.890005    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-stats-default\" is forbidden: User \"system:node:ip-10-0-142-96.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-96.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret"
Sep 17 23:11:49.890559 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:49.890537    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca-bundle\" is forbidden: User \"system:node:ip-10-0-142-96.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-96.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap"
Sep 17 23:11:49.890644 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:49.890580    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-96.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-96.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 23:11:49.890644 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:49.890582    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-dockercfg-ltrmg\" is forbidden: User \"system:node:ip-10-0-142-96.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-96.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ltrmg\"" type="*v1.Secret"
Sep 17 23:11:49.890881 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:49.890864    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"router-metrics-certs-default\" is forbidden: User \"system:node:ip-10-0-142-96.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-96.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret"
Sep 17 23:11:49.890881 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:49.890862    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"router-default-5d6966fbcb-qhw8d\" is forbidden: User \"system:node:ip-10-0-142-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-96.ec2.internal' and this object" podUID="74ddd157-12ec-4990-b2c9-04382d423ae5" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:49.890881 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:49.890873    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-96.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-96.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 23:11:49.891222 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:49.891203    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-ingress-cert\" is forbidden: User \"system:node:ip-10-0-142-96.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-96.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret"
Sep 17 23:11:49.914527 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.914494    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d6966fbcb-qhw8d"]
Sep 17 23:11:49.990838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.990796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7njl\" (UniqueName: \"kubernetes.io/projected/74ddd157-12ec-4990-b2c9-04382d423ae5-kube-api-access-d7njl\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:49.991005 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.990881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:49.991005 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.990910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-default-certificate\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:49.991005 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.990931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-stats-auth\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:49.991005 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:49.990996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:50.092198 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.092149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:50.092198 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.092206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7njl\" (UniqueName: \"kubernetes.io/projected/74ddd157-12ec-4990-b2c9-04382d423ae5-kube-api-access-d7njl\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:50.092305 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.092248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:50.092305 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.092272    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-default-certificate\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:50.092369 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.092306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-stats-auth\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:50.695277 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.695244    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 23:11:50.702650 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:50.702619    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:51.202597318 +0000 UTC m=+74.137043062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : configmap references non-existent config key: service-ca.crt
Sep 17 23:11:50.847012 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.846977    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ltrmg\""
Sep 17 23:11:50.870589 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.870559    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 23:11:50.874952 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.874926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-stats-auth\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:50.899849 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:50.899822    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 23:11:51.093065 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:51.092928    2583 secret.go:189] Couldn't get secret openshift-ingress/default-ingress-cert: failed to sync secret cache: timed out waiting for the condition
Sep 17 23:11:51.093065 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:51.093015    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-default-certificate podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:51.592998645 +0000 UTC m=+74.527444378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "default-certificate" (UniqueName: "kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-default-certificate") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : failed to sync secret cache: timed out waiting for the condition
Sep 17 23:11:51.093065 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:51.092927    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: failed to sync secret cache: timed out waiting for the condition
Sep 17 23:11:51.093348 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:51.093099    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:51.593084498 +0000 UTC m=+74.527530238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : failed to sync secret cache: timed out waiting for the condition
Sep 17 23:11:51.110072 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:51.110041    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 23:11:51.298299 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:51.298248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:51.298472 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:51.298436    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:52.29841721 +0000 UTC m=+75.232862961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : configmap references non-existent config key: service-ca.crt
Sep 17 23:11:51.421235 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:51.421157    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 23:11:51.429171 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:51.429144    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 23:11:51.429581 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:51.429561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7njl\" (UniqueName: \"kubernetes.io/projected/74ddd157-12ec-4990-b2c9-04382d423ae5-kube-api-access-d7njl\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:51.600814 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:51.600772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:51.600814 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:51.600819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-default-certificate\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:51.601058 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:51.600925    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:11:51.601058 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:51.600996    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:52.600979151 +0000 UTC m=+75.535424879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : secret "router-metrics-certs-default" not found
Sep 17 23:11:51.603148 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:51.603126    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-default-certificate\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:52.305968 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:52.305933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:52.306362 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:52.306097    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:54.306081133 +0000 UTC m=+77.240526861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : configmap references non-existent config key: service-ca.crt
Sep 17 23:11:52.607557 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:52.607456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:52.607714 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:52.607622    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:11:52.607714 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:52.607693    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:54.607678058 +0000 UTC m=+77.542123786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : secret "router-metrics-certs-default" not found
Sep 17 23:11:54.119073 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.119037    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m8dj7"]
Sep 17 23:11:54.122184 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.122164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.125505 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.125474    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q55mc\""
Sep 17 23:11:54.125656 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.125506    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:11:54.125656 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.125520    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:11:54.126280 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.126263    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:11:54.126580 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.126565    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:11:54.136789 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.136763    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m8dj7"]
Sep 17 23:11:54.218709 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.218663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/786a604b-77c4-4ab1-995f-518694bc2b34-data-volume\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.218909 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.218716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.218909 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.218760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ml26\" (UniqueName: \"kubernetes.io/projected/786a604b-77c4-4ab1-995f-518694bc2b34-kube-api-access-5ml26\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.218909 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.218796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.219046 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.218921    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/786a604b-77c4-4ab1-995f-518694bc2b34-crio-socket\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.319664 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.319622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.319812 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.319680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:54.319812 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.319714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ml26\" (UniqueName: \"kubernetes.io/projected/786a604b-77c4-4ab1-995f-518694bc2b34-kube-api-access-5ml26\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.319812 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.319748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.319812 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.319781    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:54.319988 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.319855    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:54.819837904 +0000 UTC m=+77.754283647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:54.319988 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.319979    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:58.319962391 +0000 UTC m=+81.254408134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : configmap references non-existent config key: service-ca.crt
Sep 17 23:11:54.320138 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.319983    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:54.320138 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.320016    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:54.82000933 +0000 UTC m=+77.754455061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:54.320138 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.320099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/786a604b-77c4-4ab1-995f-518694bc2b34-crio-socket\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.320384 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.320140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/786a604b-77c4-4ab1-995f-518694bc2b34-data-volume\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.320384 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.320209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/786a604b-77c4-4ab1-995f-518694bc2b34-crio-socket\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.320449 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.320409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/786a604b-77c4-4ab1-995f-518694bc2b34-data-volume\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.332301 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.332271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ml26\" (UniqueName: \"kubernetes.io/projected/786a604b-77c4-4ab1-995f-518694bc2b34-kube-api-access-5ml26\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.622118 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.622085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:54.622306 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.622206    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:11:54.622306 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.622258    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:58.622245117 +0000 UTC m=+81.556690844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : secret "router-metrics-certs-default" not found
Sep 17 23:11:54.823562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.823525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.823764 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:54.823603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:54.823764 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.823666    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:54.823764 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.823686    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:54.823764 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.823758    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:55.823738478 +0000 UTC m=+78.758184209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:54.823980 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:54.823773    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:55.82376545 +0000 UTC m=+78.758211178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:55.831114 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:55.831061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:55.831617 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:55.831169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:55.831617 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:55.831194    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:55.831617 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:55.831266    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:55.831617 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:55.831280    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:57.831257633 +0000 UTC m=+80.765703360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:55.831617 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:55.831309    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:57.831297392 +0000 UTC m=+80.765743141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:56.234958 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:56.234875    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dlnfn_1aced041-ad96-4073-8c9a-dc28c37ea129/dns-node-resolver/0.log"
Sep 17 23:11:57.032228 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:57.032201    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x9277_fb5aa21c-1af6-47e6-b02e-1417aecdc5ad/node-ca/0.log"
Sep 17 23:11:57.847991 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:57.847956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:57.848223 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:57.848040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:11:57.848223 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:57.848121    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:57.848223 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:57.848192    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:01.848177546 +0000 UTC m=+84.782623293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:11:57.848223 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:57.848130    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:11:57.848407 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:57.848286    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:01.848268355 +0000 UTC m=+84.782714088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : configmap "kube-rbac-proxy" not found
Sep 17 23:11:57.859037 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:57.858999    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r5r48_79530f25-f88f-4dfa-8077-90c9a35211dd/serve-healthcheck-canary/0.log"
Sep 17 23:11:57.996283 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:57.996248    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"]
Sep 17 23:11:57.999421 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:57.999403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:11:58.004154 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.004132    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 23:11:58.004877 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.004845    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dn7j9\""
Sep 17 23:11:58.004999 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.004883    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 23:11:58.004999 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.004931    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:11:58.020152 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.020120    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"]
Sep 17 23:11:58.050203 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.050164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2tz\" (UniqueName: \"kubernetes.io/projected/5e4ae8e6-94dc-4240-84d5-530faff388dc-kube-api-access-pz2tz\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:11:58.050203 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.050211    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:11:58.151490 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.151405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2tz\" (UniqueName: \"kubernetes.io/projected/5e4ae8e6-94dc-4240-84d5-530faff388dc-kube-api-access-pz2tz\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:11:58.151490 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.151446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:11:58.151699 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:58.151561    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:11:58.151699 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:58.151613    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls podName:5e4ae8e6-94dc-4240-84d5-530faff388dc nodeName:}" failed. No retries permitted until 2026-09-17 23:11:58.651599282 +0000 UTC m=+81.586045023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wngnl" (UID: "5e4ae8e6-94dc-4240-84d5-530faff388dc") : secret "samples-operator-tls" not found
Sep 17 23:11:58.161865 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.161831    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2tz\" (UniqueName: \"kubernetes.io/projected/5e4ae8e6-94dc-4240-84d5-530faff388dc-kube-api-access-pz2tz\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:11:58.353586 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.353543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:58.353767 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:58.353731    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:06.353712934 +0000 UTC m=+89.288158666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : configmap references non-existent config key: service-ca.crt
Sep 17 23:11:58.657132 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.657082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:11:58.657132 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.657153    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:11:58.657358 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:58.657265    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:11:58.657358 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:58.657266    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:11:58.657358 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:58.657320    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:06.657305017 +0000 UTC m=+89.591750745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : secret "router-metrics-certs-default" not found
Sep 17 23:11:58.657358 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:58.657333    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls podName:5e4ae8e6-94dc-4240-84d5-530faff388dc nodeName:}" failed. No retries permitted until 2026-09-17 23:11:59.657328222 +0000 UTC m=+82.591773951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wngnl" (UID: "5e4ae8e6-94dc-4240-84d5-530faff388dc") : secret "samples-operator-tls" not found
Sep 17 23:11:58.740591 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.740556    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-g5c6q"]
Sep 17 23:11:58.743429 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.743411    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.746557 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.746538    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 23:11:58.747657 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.747637    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-77cld\""
Sep 17 23:11:58.749078 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.749060    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:11:58.749154 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.749110    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 23:11:58.761312 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.761280    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 23:11:58.788210 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.788177    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-g5c6q"]
Sep 17 23:11:58.788691 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.788671    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 23:11:58.858700 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.858666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a31ae514-8497-4e88-89e7-fa894e336f26-serving-cert\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.858801 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.858700    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4xl\" (UniqueName: \"kubernetes.io/projected/a31ae514-8497-4e88-89e7-fa894e336f26-kube-api-access-zq4xl\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.858801 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.858724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a31ae514-8497-4e88-89e7-fa894e336f26-config\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.858865 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.858836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a31ae514-8497-4e88-89e7-fa894e336f26-trusted-ca\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.959906 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.959817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a31ae514-8497-4e88-89e7-fa894e336f26-serving-cert\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.959906 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.959856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4xl\" (UniqueName: \"kubernetes.io/projected/a31ae514-8497-4e88-89e7-fa894e336f26-kube-api-access-zq4xl\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.959906 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.959878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a31ae514-8497-4e88-89e7-fa894e336f26-config\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.960166 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.959924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a31ae514-8497-4e88-89e7-fa894e336f26-trusted-ca\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.960607 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.960583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a31ae514-8497-4e88-89e7-fa894e336f26-config\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.960686 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.960674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a31ae514-8497-4e88-89e7-fa894e336f26-trusted-ca\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.962341 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.962320    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a31ae514-8497-4e88-89e7-fa894e336f26-serving-cert\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:58.970327 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:58.970302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4xl\" (UniqueName: \"kubernetes.io/projected/a31ae514-8497-4e88-89e7-fa894e336f26-kube-api-access-zq4xl\") pod \"console-operator-fd4bb46c-g5c6q\" (UID: \"a31ae514-8497-4e88-89e7-fa894e336f26\") " pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:59.053609 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:59.053563    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:11:59.329347 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:59.329313    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-g5c6q"]
Sep 17 23:11:59.332875 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:11:59.332827    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31ae514_8497_4e88_89e7_fa894e336f26.slice/crio-218aca830df73bf48f4b190b26557c4e1c170c22876d9e332a879219199ad45f WatchSource:0}: Error finding container 218aca830df73bf48f4b190b26557c4e1c170c22876d9e332a879219199ad45f: Status 404 returned error can't find the container with id 218aca830df73bf48f4b190b26557c4e1c170c22876d9e332a879219199ad45f
Sep 17 23:11:59.496457 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:59.496420    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" event={"ID":"a31ae514-8497-4e88-89e7-fa894e336f26","Type":"ContainerStarted","Data":"218aca830df73bf48f4b190b26557c4e1c170c22876d9e332a879219199ad45f"}
Sep 17 23:11:59.666679 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:11:59.666571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:11:59.666842 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:59.666719    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:11:59.666842 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:11:59.666776    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls podName:5e4ae8e6-94dc-4240-84d5-530faff388dc nodeName:}" failed. No retries permitted until 2026-09-17 23:12:01.666761236 +0000 UTC m=+84.601206967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wngnl" (UID: "5e4ae8e6-94dc-4240-84d5-530faff388dc") : secret "samples-operator-tls" not found
Sep 17 23:12:00.001218 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.001136    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"]
Sep 17 23:12:00.004234 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.004210    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.008597 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.008570    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 23:12:00.009049 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.008631    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 23:12:00.009659 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.009636    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 23:12:00.012958 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.012935    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:00.013978 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.013954    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vl7xt\""
Sep 17 23:12:00.020236 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.020211    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"]
Sep 17 23:12:00.070353 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.070315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1c623b6-ba24-484f-bdc0-06dd1a93f1b2-serving-cert\") pod \"service-ca-operator-57cdc4b768-6hmnf\" (UID: \"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.070769 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.070387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1c623b6-ba24-484f-bdc0-06dd1a93f1b2-config\") pod \"service-ca-operator-57cdc4b768-6hmnf\" (UID: \"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.070769 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.070445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwj7h\" (UniqueName: \"kubernetes.io/projected/d1c623b6-ba24-484f-bdc0-06dd1a93f1b2-kube-api-access-kwj7h\") pod \"service-ca-operator-57cdc4b768-6hmnf\" (UID: \"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.171635 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.171598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1c623b6-ba24-484f-bdc0-06dd1a93f1b2-serving-cert\") pod \"service-ca-operator-57cdc4b768-6hmnf\" (UID: \"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.171808 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.171665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1c623b6-ba24-484f-bdc0-06dd1a93f1b2-config\") pod \"service-ca-operator-57cdc4b768-6hmnf\" (UID: \"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.171808 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.171719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwj7h\" (UniqueName: \"kubernetes.io/projected/d1c623b6-ba24-484f-bdc0-06dd1a93f1b2-kube-api-access-kwj7h\") pod \"service-ca-operator-57cdc4b768-6hmnf\" (UID: \"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.172318 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.172286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1c623b6-ba24-484f-bdc0-06dd1a93f1b2-config\") pod \"service-ca-operator-57cdc4b768-6hmnf\" (UID: \"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.174261 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.174237    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1c623b6-ba24-484f-bdc0-06dd1a93f1b2-serving-cert\") pod \"service-ca-operator-57cdc4b768-6hmnf\" (UID: \"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.181883 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.181851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwj7h\" (UniqueName: \"kubernetes.io/projected/d1c623b6-ba24-484f-bdc0-06dd1a93f1b2-kube-api-access-kwj7h\") pod \"service-ca-operator-57cdc4b768-6hmnf\" (UID: \"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.315301 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.315259    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"
Sep 17 23:12:00.601200 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:00.601125    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf"]
Sep 17 23:12:01.038135 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:01.038100    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c623b6_ba24_484f_bdc0_06dd1a93f1b2.slice/crio-56915c8ddbe752d11c56309a0193abd9aca1cc3e53008ad0b05ede03f61690f2 WatchSource:0}: Error finding container 56915c8ddbe752d11c56309a0193abd9aca1cc3e53008ad0b05ede03f61690f2: Status 404 returned error can't find the container with id 56915c8ddbe752d11c56309a0193abd9aca1cc3e53008ad0b05ede03f61690f2
Sep 17 23:12:01.504277 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:01.504197    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/0.log"
Sep 17 23:12:01.504277 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:01.504242    2583 generic.go:356] "Generic (PLEG): container finished" podID="a31ae514-8497-4e88-89e7-fa894e336f26" containerID="fdab6c4d977c8387dd0cd5039cf9c1301e444863aa588d4210057cc20a197606" exitCode=255
Sep 17 23:12:01.504751 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:01.504318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" event={"ID":"a31ae514-8497-4e88-89e7-fa894e336f26","Type":"ContainerDied","Data":"fdab6c4d977c8387dd0cd5039cf9c1301e444863aa588d4210057cc20a197606"}
Sep 17 23:12:01.504751 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:01.504535    2583 scope.go:117] "RemoveContainer" containerID="fdab6c4d977c8387dd0cd5039cf9c1301e444863aa588d4210057cc20a197606"
Sep 17 23:12:01.505684 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:01.505661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf" event={"ID":"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2","Type":"ContainerStarted","Data":"56915c8ddbe752d11c56309a0193abd9aca1cc3e53008ad0b05ede03f61690f2"}
Sep 17 23:12:01.688035 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:01.687985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:12:01.688220 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:01.688186    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:12:01.688310 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:01.688298    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls podName:5e4ae8e6-94dc-4240-84d5-530faff388dc nodeName:}" failed. No retries permitted until 2026-09-17 23:12:05.688272885 +0000 UTC m=+88.622718637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wngnl" (UID: "5e4ae8e6-94dc-4240-84d5-530faff388dc") : secret "samples-operator-tls" not found
Sep 17 23:12:01.890084 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:01.890039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:12:01.890272 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:01.890113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:12:01.890272 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:01.890227    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:01.890272 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:01.890243    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:01.890440 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:01.890317    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:09.890293373 +0000 UTC m=+92.824739102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:01.890440 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:01.890344    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:09.890331248 +0000 UTC m=+92.824776994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:02.511499 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:02.511464    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/1.log"
Sep 17 23:12:02.511974 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:02.511957    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/0.log"
Sep 17 23:12:02.512048 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:02.511997    2583 generic.go:356] "Generic (PLEG): container finished" podID="a31ae514-8497-4e88-89e7-fa894e336f26" containerID="3ed30df021b985875f883aa090704a326ab8d524460e552f453e63567775579f" exitCode=255
Sep 17 23:12:02.512100 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:02.512054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" event={"ID":"a31ae514-8497-4e88-89e7-fa894e336f26","Type":"ContainerDied","Data":"3ed30df021b985875f883aa090704a326ab8d524460e552f453e63567775579f"}
Sep 17 23:12:02.512133 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:02.512116    2583 scope.go:117] "RemoveContainer" containerID="fdab6c4d977c8387dd0cd5039cf9c1301e444863aa588d4210057cc20a197606"
Sep 17 23:12:02.512398 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:02.512379    2583 scope.go:117] "RemoveContainer" containerID="3ed30df021b985875f883aa090704a326ab8d524460e552f453e63567775579f"
Sep 17 23:12:02.512598 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:02.512567    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-g5c6q_openshift-console-operator(a31ae514-8497-4e88-89e7-fa894e336f26)\"" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" podUID="a31ae514-8497-4e88-89e7-fa894e336f26"
Sep 17 23:12:03.517259 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:03.517215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf" event={"ID":"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2","Type":"ContainerStarted","Data":"ce1f626d8e56f570bc89cb12ca347c22f904f2012160d414bef37c5e2489d502"}
Sep 17 23:12:03.520855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:03.520831    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/1.log"
Sep 17 23:12:03.521280 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:03.521260    2583 scope.go:117] "RemoveContainer" containerID="3ed30df021b985875f883aa090704a326ab8d524460e552f453e63567775579f"
Sep 17 23:12:03.521465 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:03.521444    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-g5c6q_openshift-console-operator(a31ae514-8497-4e88-89e7-fa894e336f26)\"" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" podUID="a31ae514-8497-4e88-89e7-fa894e336f26"
Sep 17 23:12:03.561895 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:03.558839    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf" podStartSLOduration=2.7654305839999997 podStartE2EDuration="4.558816913s" totalImagesPullingTime="1.793386329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:03.535295611 +0000 UTC m=+86.469741363" watchObservedRunningTime="2026-09-17 23:12:03.558816913 +0000 UTC m=+86.493262665"
Sep 17 23:12:04.522396 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:04.522345    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj"]
Sep 17 23:12:04.525811 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:04.525787    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj"
Sep 17 23:12:04.528322 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:04.528298    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cjdvj\""
Sep 17 23:12:04.534692 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:04.534667    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj"]
Sep 17 23:12:04.614145 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:04.614099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krxq7\" (UniqueName: \"kubernetes.io/projected/ff234264-6281-402d-8c44-e5a685191ae3-kube-api-access-krxq7\") pod \"network-check-source-7cbc45fddc-rkzvj\" (UID: \"ff234264-6281-402d-8c44-e5a685191ae3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj"
Sep 17 23:12:04.715131 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:04.715096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krxq7\" (UniqueName: \"kubernetes.io/projected/ff234264-6281-402d-8c44-e5a685191ae3-kube-api-access-krxq7\") pod \"network-check-source-7cbc45fddc-rkzvj\" (UID: \"ff234264-6281-402d-8c44-e5a685191ae3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj"
Sep 17 23:12:04.727840 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:04.727804    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krxq7\" (UniqueName: \"kubernetes.io/projected/ff234264-6281-402d-8c44-e5a685191ae3-kube-api-access-krxq7\") pod \"network-check-source-7cbc45fddc-rkzvj\" (UID: \"ff234264-6281-402d-8c44-e5a685191ae3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj"
Sep 17 23:12:04.834933 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:04.834850    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj"
Sep 17 23:12:05.114803 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:05.114774    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj"]
Sep 17 23:12:05.116371 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:05.116342    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff234264_6281_402d_8c44_e5a685191ae3.slice/crio-36c0dcbc098d86c02293d1515233614e5e9ea0bc269f6db391963222973888a9 WatchSource:0}: Error finding container 36c0dcbc098d86c02293d1515233614e5e9ea0bc269f6db391963222973888a9: Status 404 returned error can't find the container with id 36c0dcbc098d86c02293d1515233614e5e9ea0bc269f6db391963222973888a9
Sep 17 23:12:05.528874 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:05.528838    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj" event={"ID":"ff234264-6281-402d-8c44-e5a685191ae3","Type":"ContainerStarted","Data":"99debf963b30a598a4becab1629458ed56829341723f4a746f2d6297f3b1fff9"}
Sep 17 23:12:05.529289 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:05.528902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj" event={"ID":"ff234264-6281-402d-8c44-e5a685191ae3","Type":"ContainerStarted","Data":"36c0dcbc098d86c02293d1515233614e5e9ea0bc269f6db391963222973888a9"}
Sep 17 23:12:05.553592 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:05.553532    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-rkzvj" podStartSLOduration=1.553510505 podStartE2EDuration="1.553510505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:05.552922931 +0000 UTC m=+88.487368677" watchObservedRunningTime="2026-09-17 23:12:05.553510505 +0000 UTC m=+88.487956256"
Sep 17 23:12:05.722869 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:05.722831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:12:05.723073 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:05.722966    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 23:12:05.723073 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:05.723048    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls podName:5e4ae8e6-94dc-4240-84d5-530faff388dc nodeName:}" failed. No retries permitted until 2026-09-17 23:12:13.723009987 +0000 UTC m=+96.657455718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-wngnl" (UID: "5e4ae8e6-94dc-4240-84d5-530faff388dc") : secret "samples-operator-tls" not found
Sep 17 23:12:06.180095 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.180060    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb"]
Sep 17 23:12:06.183258 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.183240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb"
Sep 17 23:12:06.186046 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.186005    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 23:12:06.187049 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.187008    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xp7df\""
Sep 17 23:12:06.187137 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.187010    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:06.194119 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.194089    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb"]
Sep 17 23:12:06.329216 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.329181    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9mc9\" (UniqueName: \"kubernetes.io/projected/a1c16b4a-7e52-4cd9-9580-82a108f0cb75-kube-api-access-t9mc9\") pod \"migrator-76794cbd6c-qzpxb\" (UID: \"a1c16b4a-7e52-4cd9-9580-82a108f0cb75\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb"
Sep 17 23:12:06.429785 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.429745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:06.429988 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.429833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9mc9\" (UniqueName: \"kubernetes.io/projected/a1c16b4a-7e52-4cd9-9580-82a108f0cb75-kube-api-access-t9mc9\") pod \"migrator-76794cbd6c-qzpxb\" (UID: \"a1c16b4a-7e52-4cd9-9580-82a108f0cb75\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb"
Sep 17 23:12:06.429988 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:06.429929    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:22.429910762 +0000 UTC m=+105.364356513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : configmap references non-existent config key: service-ca.crt
Sep 17 23:12:06.438726 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.438677    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9mc9\" (UniqueName: \"kubernetes.io/projected/a1c16b4a-7e52-4cd9-9580-82a108f0cb75-kube-api-access-t9mc9\") pod \"migrator-76794cbd6c-qzpxb\" (UID: \"a1c16b4a-7e52-4cd9-9580-82a108f0cb75\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb"
Sep 17 23:12:06.491907 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.491867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb"
Sep 17 23:12:06.731714 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.731623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:06.732211 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:06.731796    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 23:12:06.732211 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:06.731885    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs podName:74ddd157-12ec-4990-b2c9-04382d423ae5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:22.731861795 +0000 UTC m=+105.666307526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs") pod "router-default-5d6966fbcb-qhw8d" (UID: "74ddd157-12ec-4990-b2c9-04382d423ae5") : secret "router-metrics-certs-default" not found
Sep 17 23:12:06.774596 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:06.774566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb"]
Sep 17 23:12:06.776956 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:06.776929    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c16b4a_7e52_4cd9_9580_82a108f0cb75.slice/crio-f06c5c7e33d9df2e642eeca531cfd89cab89dfd3d881b380425d29e37c1b2775 WatchSource:0}: Error finding container f06c5c7e33d9df2e642eeca531cfd89cab89dfd3d881b380425d29e37c1b2775: Status 404 returned error can't find the container with id f06c5c7e33d9df2e642eeca531cfd89cab89dfd3d881b380425d29e37c1b2775
Sep 17 23:12:07.541264 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:07.541225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb" event={"ID":"a1c16b4a-7e52-4cd9-9580-82a108f0cb75","Type":"ContainerStarted","Data":"f06c5c7e33d9df2e642eeca531cfd89cab89dfd3d881b380425d29e37c1b2775"}
Sep 17 23:12:08.546818 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:08.546783    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb" event={"ID":"a1c16b4a-7e52-4cd9-9580-82a108f0cb75","Type":"ContainerStarted","Data":"162cc053f1e61a852c0203ec2d1ea756f9d4dd410a48432cf0605e1fe262f633"}
Sep 17 23:12:08.547333 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:08.546834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb" event={"ID":"a1c16b4a-7e52-4cd9-9580-82a108f0cb75","Type":"ContainerStarted","Data":"2d991e01d71c68f5e9d7d66c602f7c0b27a8853f673f5c0d16017f004757f0be"}
Sep 17 23:12:09.054000 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:09.053956    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:12:09.054210 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:09.054039    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:12:09.054403 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:09.054390    2583 scope.go:117] "RemoveContainer" containerID="3ed30df021b985875f883aa090704a326ab8d524460e552f453e63567775579f"
Sep 17 23:12:09.054551 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:09.054530    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-g5c6q_openshift-console-operator(a31ae514-8497-4e88-89e7-fa894e336f26)\"" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" podUID="a31ae514-8497-4e88-89e7-fa894e336f26"
Sep 17 23:12:09.960489 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:09.960425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:12:09.960489 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:09.960497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:12:09.960943 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:09.960590    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:09.960943 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:09.960660    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls podName:786a604b-77c4-4ab1-995f-518694bc2b34 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:25.960641955 +0000 UTC m=+108.895087704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m8dj7" (UID: "786a604b-77c4-4ab1-995f-518694bc2b34") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:09.961093 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:09.961075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/786a604b-77c4-4ab1-995f-518694bc2b34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:12:13.794586 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:13.794527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:12:13.797123 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:13.797096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e4ae8e6-94dc-4240-84d5-530faff388dc-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-wngnl\" (UID: \"5e4ae8e6-94dc-4240-84d5-530faff388dc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:12:13.908555 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:13.908500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"
Sep 17 23:12:14.189567 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:14.189470    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-qzpxb" podStartSLOduration=7.167956177 podStartE2EDuration="8.189452027s" totalImagesPullingTime="1.02149585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:08.580224896 +0000 UTC m=+91.514670646" watchObservedRunningTime="2026-09-17 23:12:14.189452027 +0000 UTC m=+97.123897833"
Sep 17 23:12:14.189718 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:14.189676    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl"]
Sep 17 23:12:14.565966 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:14.565924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl" event={"ID":"5e4ae8e6-94dc-4240-84d5-530faff388dc","Type":"ContainerStarted","Data":"829c59fc423967ac5515c9d95f07dec900817311ba96d6fe154e6ec4a1179e1d"}
Sep 17 23:12:16.516011 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.515965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:12:16.518432 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.518413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"image-registry-74dcc75879-rt6s8\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") " pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:12:16.533885 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.533853    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6gj7c\""
Sep 17 23:12:16.541860 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.541838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:12:16.575518 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.575479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl" event={"ID":"5e4ae8e6-94dc-4240-84d5-530faff388dc","Type":"ContainerStarted","Data":"d89b8638fa6db762e4edbd30e4217b9a936962b074b484b776aff7f89f4a3783"}
Sep 17 23:12:16.575741 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.575557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl" event={"ID":"5e4ae8e6-94dc-4240-84d5-530faff388dc","Type":"ContainerStarted","Data":"4d6ac7817a3c876b1fdd9ee0ff5e893348945b657c43f8352fbec84801773111"}
Sep 17 23:12:16.617192 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.616689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:12:16.622454 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.622421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f200acdd-f4e1-4afa-a742-4d5981e022cb-metrics-tls\") pod \"dns-default-548p9\" (UID: \"f200acdd-f4e1-4afa-a742-4d5981e022cb\") " pod="openshift-dns/dns-default-548p9"
Sep 17 23:12:16.861872 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.861685    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-wngnl" podStartSLOduration=18.36960424 podStartE2EDuration="19.861665255s" totalImagesPullingTime="1.492061015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:16.605871618 +0000 UTC m=+99.540317374" watchObservedRunningTime="2026-09-17 23:12:16.861665255 +0000 UTC m=+99.796111021"
Sep 17 23:12:16.862918 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.862897    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74dcc75879-rt6s8"]
Sep 17 23:12:16.866339 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:16.866302    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09eaea72_e75d_4676_8fc0_e528b568a81f.slice/crio-543c872a7dbe01e0525a25a2f6653d457b3b1888fe97c967777caff21c040cad WatchSource:0}: Error finding container 543c872a7dbe01e0525a25a2f6653d457b3b1888fe97c967777caff21c040cad: Status 404 returned error can't find the container with id 543c872a7dbe01e0525a25a2f6653d457b3b1888fe97c967777caff21c040cad
Sep 17 23:12:16.884083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.884057    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xxcjw\""
Sep 17 23:12:16.891402 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:16.891367    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-548p9"
Sep 17 23:12:17.179455 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:17.179421    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-548p9"]
Sep 17 23:12:17.184763 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:17.184730    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf200acdd_f4e1_4afa_a742_4d5981e022cb.slice/crio-27f848cf5e2aca4b782a221ad54d0abb3cc0e6c9608ae680185fadcc3a2cd874 WatchSource:0}: Error finding container 27f848cf5e2aca4b782a221ad54d0abb3cc0e6c9608ae680185fadcc3a2cd874: Status 404 returned error can't find the container with id 27f848cf5e2aca4b782a221ad54d0abb3cc0e6c9608ae680185fadcc3a2cd874
Sep 17 23:12:17.580349 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:17.580310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8" event={"ID":"09eaea72-e75d-4676-8fc0-e528b568a81f","Type":"ContainerStarted","Data":"28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e"}
Sep 17 23:12:17.580820 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:17.580421    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:12:17.580820 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:17.580447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8" event={"ID":"09eaea72-e75d-4676-8fc0-e528b568a81f","Type":"ContainerStarted","Data":"543c872a7dbe01e0525a25a2f6653d457b3b1888fe97c967777caff21c040cad"}
Sep 17 23:12:17.581472 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:17.581444    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-548p9" event={"ID":"f200acdd-f4e1-4afa-a742-4d5981e022cb","Type":"ContainerStarted","Data":"27f848cf5e2aca4b782a221ad54d0abb3cc0e6c9608ae680185fadcc3a2cd874"}
Sep 17 23:12:19.593656 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:19.593616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-548p9" event={"ID":"f200acdd-f4e1-4afa-a742-4d5981e022cb","Type":"ContainerStarted","Data":"d7c22cf8115046b7665c9092e14bc2b5e4a30c40e0c08159cb73ae3ede6784f7"}
Sep 17 23:12:19.594114 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:19.593705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-548p9" event={"ID":"f200acdd-f4e1-4afa-a742-4d5981e022cb","Type":"ContainerStarted","Data":"4e8e4cf9e8e403c46e2b9623640fe7ab2cc85f045af61c56ca75a1f6f3b9992d"}
Sep 17 23:12:19.594114 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:19.593734    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-548p9"
Sep 17 23:12:19.618086 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:19.618016    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8" podStartSLOduration=100.617999564 podStartE2EDuration="1m40.617999564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:17.628652876 +0000 UTC m=+100.563098628" watchObservedRunningTime="2026-09-17 23:12:19.617999564 +0000 UTC m=+102.552445314"
Sep 17 23:12:19.618318 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:19.618299    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-548p9" podStartSLOduration=65.934583177 podStartE2EDuration="1m7.618293724s" totalImagesPullingTime="1.683710547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:19.616223334 +0000 UTC m=+102.550669084" watchObservedRunningTime="2026-09-17 23:12:19.618293724 +0000 UTC m=+102.552739474"
Sep 17 23:12:20.473965 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:20.473932    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b224x"
Sep 17 23:12:22.466673 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:22.466626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:22.467485 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:22.467462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74ddd157-12ec-4990-b2c9-04382d423ae5-service-ca-bundle\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:22.769379 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:22.769344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:22.771825 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:22.771799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ddd157-12ec-4990-b2c9-04382d423ae5-metrics-certs\") pod \"router-default-5d6966fbcb-qhw8d\" (UID: \"74ddd157-12ec-4990-b2c9-04382d423ae5\") " pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:22.890279 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:22.890228    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:23.166897 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:23.166858    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d6966fbcb-qhw8d"]
Sep 17 23:12:23.169357 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:23.169331    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ddd157_12ec_4990_b2c9_04382d423ae5.slice/crio-8dcea39ee26d92cc9499c0ff959339137f8073fa9946b7ce4e99477032d0402e WatchSource:0}: Error finding container 8dcea39ee26d92cc9499c0ff959339137f8073fa9946b7ce4e99477032d0402e: Status 404 returned error can't find the container with id 8dcea39ee26d92cc9499c0ff959339137f8073fa9946b7ce4e99477032d0402e
Sep 17 23:12:23.608080 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:23.608044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d" event={"ID":"74ddd157-12ec-4990-b2c9-04382d423ae5","Type":"ContainerStarted","Data":"25787ef2cae920323065a44aec06502e258c53e204292dbf3ecddfec6abf5ee4"}
Sep 17 23:12:23.608499 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:23.608100    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d" event={"ID":"74ddd157-12ec-4990-b2c9-04382d423ae5","Type":"ContainerStarted","Data":"8dcea39ee26d92cc9499c0ff959339137f8073fa9946b7ce4e99477032d0402e"}
Sep 17 23:12:23.631849 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:23.631785    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d" podStartSLOduration=34.631768109 podStartE2EDuration="34.631768109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:23.631114097 +0000 UTC m=+106.565559848" watchObservedRunningTime="2026-09-17 23:12:23.631768109 +0000 UTC m=+106.566213856"
Sep 17 23:12:23.891272 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:23.891186    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:23.893850 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:23.893824    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:24.152007 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:24.151924    2583 scope.go:117] "RemoveContainer" containerID="3ed30df021b985875f883aa090704a326ab8d524460e552f453e63567775579f"
Sep 17 23:12:24.612644 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:24.612613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:12:24.613125 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:24.613109    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/1.log"
Sep 17 23:12:24.613208 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:24.613148    2583 generic.go:356] "Generic (PLEG): container finished" podID="a31ae514-8497-4e88-89e7-fa894e336f26" containerID="3e19e47fc2858990447376a02ea0fafbab4dfa71e10460c88a141c83f4ec8260" exitCode=255
Sep 17 23:12:24.613269 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:24.613226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" event={"ID":"a31ae514-8497-4e88-89e7-fa894e336f26","Type":"ContainerDied","Data":"3e19e47fc2858990447376a02ea0fafbab4dfa71e10460c88a141c83f4ec8260"}
Sep 17 23:12:24.613331 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:24.613285    2583 scope.go:117] "RemoveContainer" containerID="3ed30df021b985875f883aa090704a326ab8d524460e552f453e63567775579f"
Sep 17 23:12:24.613485 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:24.613469    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:24.613801 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:24.613782    2583 scope.go:117] "RemoveContainer" containerID="3e19e47fc2858990447376a02ea0fafbab4dfa71e10460c88a141c83f4ec8260"
Sep 17 23:12:24.613978 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:24.613947    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-g5c6q_openshift-console-operator(a31ae514-8497-4e88-89e7-fa894e336f26)\"" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" podUID="a31ae514-8497-4e88-89e7-fa894e336f26"
Sep 17 23:12:24.614923 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:24.614905    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5d6966fbcb-qhw8d"
Sep 17 23:12:25.618160 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:25.618128    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:12:25.996067 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:25.995926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:12:25.998350 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:25.998315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/786a604b-77c4-4ab1-995f-518694bc2b34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8dj7\" (UID: \"786a604b-77c4-4ab1-995f-518694bc2b34\") " pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:12:26.231952 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:26.231912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m8dj7"
Sep 17 23:12:26.512759 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:26.512733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m8dj7"]
Sep 17 23:12:26.515139 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:26.515108    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786a604b_77c4_4ab1_995f_518694bc2b34.slice/crio-a823019d002a40aad6a70e7e88d0773d71836c3e4b5ecb851551304a9c5c1480 WatchSource:0}: Error finding container a823019d002a40aad6a70e7e88d0773d71836c3e4b5ecb851551304a9c5c1480: Status 404 returned error can't find the container with id a823019d002a40aad6a70e7e88d0773d71836c3e4b5ecb851551304a9c5c1480
Sep 17 23:12:26.623475 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:26.623439    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m8dj7" event={"ID":"786a604b-77c4-4ab1-995f-518694bc2b34","Type":"ContainerStarted","Data":"b3139ac98c8b6bd6c34eea2bf85d746cb072f14d7a721a7b5f65662bac784cde"}
Sep 17 23:12:26.623851 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:26.623494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m8dj7" event={"ID":"786a604b-77c4-4ab1-995f-518694bc2b34","Type":"ContainerStarted","Data":"a823019d002a40aad6a70e7e88d0773d71836c3e4b5ecb851551304a9c5c1480"}
Sep 17 23:12:27.629567 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:27.629527    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m8dj7" event={"ID":"786a604b-77c4-4ab1-995f-518694bc2b34","Type":"ContainerStarted","Data":"7240f87f57d6e7eabd5737eabd223e0c979869e61b19a41dbbc621c8465237a7"}
Sep 17 23:12:28.634581 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:28.634541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m8dj7" event={"ID":"786a604b-77c4-4ab1-995f-518694bc2b34","Type":"ContainerStarted","Data":"1932f653bf1c1378e0d4d73a192e25c1476b56f872a2968a89d14d240fd25d2f"}
Sep 17 23:12:28.659991 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:28.659934    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m8dj7" podStartSLOduration=32.878399096 podStartE2EDuration="34.659914281s" totalImagesPullingTime="1.781515185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:28.659802917 +0000 UTC m=+111.594248678" watchObservedRunningTime="2026-09-17 23:12:28.659914281 +0000 UTC m=+111.594360031"
Sep 17 23:12:29.054652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:29.054609    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:12:29.054838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:29.054710    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:12:29.054976 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:29.054963    2583 scope.go:117] "RemoveContainer" containerID="3e19e47fc2858990447376a02ea0fafbab4dfa71e10460c88a141c83f4ec8260"
Sep 17 23:12:29.055163 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:29.055145    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-g5c6q_openshift-console-operator(a31ae514-8497-4e88-89e7-fa894e336f26)\"" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" podUID="a31ae514-8497-4e88-89e7-fa894e336f26"
Sep 17 23:12:29.599710 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:29.599678    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-548p9"
Sep 17 23:12:29.638075 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:29.638047    2583 scope.go:117] "RemoveContainer" containerID="3e19e47fc2858990447376a02ea0fafbab4dfa71e10460c88a141c83f4ec8260"
Sep 17 23:12:29.638547 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:29.638264    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-g5c6q_openshift-console-operator(a31ae514-8497-4e88-89e7-fa894e336f26)\"" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" podUID="a31ae514-8497-4e88-89e7-fa894e336f26"
Sep 17 23:12:36.546892 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:36.546848    2583 patch_prober.go:28] interesting pod/image-registry-74dcc75879-rt6s8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 23:12:36.547284 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:36.546922    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8" podUID="09eaea72-e75d-4676-8fc0-e528b568a81f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:12:38.121153 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.121117    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"]
Sep 17 23:12:38.124759 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.124738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.128735 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.128708    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 23:12:38.128735 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.128708    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-44dmk\""
Sep 17 23:12:38.129705 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.129683    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:12:38.130055 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.130036    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 23:12:38.130973 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.130956    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:12:38.135403 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.135382    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:12:38.144876 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.144839    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"]
Sep 17 23:12:38.194041 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.193977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a679c3fe-edd5-413c-9367-fee3c1a11cac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.194240 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.194086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh5sc\" (UniqueName: \"kubernetes.io/projected/a679c3fe-edd5-413c-9367-fee3c1a11cac-kube-api-access-xh5sc\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.194240 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.194135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a679c3fe-edd5-413c-9367-fee3c1a11cac-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.194240 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.194193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a679c3fe-edd5-413c-9367-fee3c1a11cac-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.295135 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.295097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh5sc\" (UniqueName: \"kubernetes.io/projected/a679c3fe-edd5-413c-9367-fee3c1a11cac-kube-api-access-xh5sc\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.295291 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.295158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a679c3fe-edd5-413c-9367-fee3c1a11cac-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.295291 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.295199    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a679c3fe-edd5-413c-9367-fee3c1a11cac-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.295291 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.295228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a679c3fe-edd5-413c-9367-fee3c1a11cac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.295444 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:38.295329    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 23:12:38.295444 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:38.295408    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a679c3fe-edd5-413c-9367-fee3c1a11cac-prometheus-operator-tls podName:a679c3fe-edd5-413c-9367-fee3c1a11cac nodeName:}" failed. No retries permitted until 2026-09-17 23:12:38.795392711 +0000 UTC m=+121.729838454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a679c3fe-edd5-413c-9367-fee3c1a11cac-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-mqbm8" (UID: "a679c3fe-edd5-413c-9367-fee3c1a11cac") : secret "prometheus-operator-tls" not found
Sep 17 23:12:38.295840 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.295819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a679c3fe-edd5-413c-9367-fee3c1a11cac-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.297663 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.297640    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a679c3fe-edd5-413c-9367-fee3c1a11cac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.305869 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.305844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh5sc\" (UniqueName: \"kubernetes.io/projected/a679c3fe-edd5-413c-9367-fee3c1a11cac-kube-api-access-xh5sc\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.589796 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.589760    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:12:38.798963 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.798903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a679c3fe-edd5-413c-9367-fee3c1a11cac-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:38.801437 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:38.801407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a679c3fe-edd5-413c-9367-fee3c1a11cac-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-mqbm8\" (UID: \"a679c3fe-edd5-413c-9367-fee3c1a11cac\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:39.036168 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:39.036129    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-44dmk\""
Sep 17 23:12:39.044454 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:39.044420    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"
Sep 17 23:12:39.332686 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:39.332609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8"]
Sep 17 23:12:39.336096 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:39.336059    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda679c3fe_edd5_413c_9367_fee3c1a11cac.slice/crio-a613e00e08e0ac9269a9aa4fb3270dc0d3cec5ac1de54a6c88f69f8fa28dd8d5 WatchSource:0}: Error finding container a613e00e08e0ac9269a9aa4fb3270dc0d3cec5ac1de54a6c88f69f8fa28dd8d5: Status 404 returned error can't find the container with id a613e00e08e0ac9269a9aa4fb3270dc0d3cec5ac1de54a6c88f69f8fa28dd8d5
Sep 17 23:12:39.673203 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:39.673114    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8" event={"ID":"a679c3fe-edd5-413c-9367-fee3c1a11cac","Type":"ContainerStarted","Data":"a613e00e08e0ac9269a9aa4fb3270dc0d3cec5ac1de54a6c88f69f8fa28dd8d5"}
Sep 17 23:12:40.152411 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:40.152378    2583 scope.go:117] "RemoveContainer" containerID="3e19e47fc2858990447376a02ea0fafbab4dfa71e10460c88a141c83f4ec8260"
Sep 17 23:12:40.152592 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:40.152571    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-g5c6q_openshift-console-operator(a31ae514-8497-4e88-89e7-fa894e336f26)\"" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" podUID="a31ae514-8497-4e88-89e7-fa894e336f26"
Sep 17 23:12:40.679864 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:40.679820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8" event={"ID":"a679c3fe-edd5-413c-9367-fee3c1a11cac","Type":"ContainerStarted","Data":"e8dd3e457d4a2a4d6f63a9e977058ab74060ca8f40ae55a2660b9034726c0e5b"}
Sep 17 23:12:40.680289 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:40.679892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8" event={"ID":"a679c3fe-edd5-413c-9367-fee3c1a11cac","Type":"ContainerStarted","Data":"47310487b616590c357e331c2f0b62956aaf148963321972bc48193534201c30"}
Sep 17 23:12:40.702953 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:40.702769    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-mqbm8" podStartSLOduration=1.60578493 podStartE2EDuration="2.702747073s" totalImagesPullingTime="1.096962143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:40.702529293 +0000 UTC m=+123.636975046" watchObservedRunningTime="2026-09-17 23:12:40.702747073 +0000 UTC m=+123.637192825"
Sep 17 23:12:42.867311 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.867279    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"]
Sep 17 23:12:42.870618 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.870598    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:42.874376 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.874350    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 23:12:42.874376 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.874356    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z84q2\""
Sep 17 23:12:42.874970 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.874953    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 23:12:42.902480 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.902445    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"]
Sep 17 23:12:42.932422 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.932382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e672a6c1-77b7-4e0a-b06d-46622f97fe40-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:42.932597 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.932440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e672a6c1-77b7-4e0a-b06d-46622f97fe40-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:42.932597 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.932467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlwsg\" (UniqueName: \"kubernetes.io/projected/e672a6c1-77b7-4e0a-b06d-46622f97fe40-kube-api-access-dlwsg\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:42.932597 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.932513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e672a6c1-77b7-4e0a-b06d-46622f97fe40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:42.977657 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.977615    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4tmfn"]
Sep 17 23:12:42.981087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.981059    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:42.983685 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.983660    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:12:42.983861 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.983719    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z85sl\""
Sep 17 23:12:42.984579 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.984556    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:12:42.984848 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:42.984829    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:12:43.033528 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/433b4c61-62dc-4448-b9a6-4ac3b681daf0-root\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.033742 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e672a6c1-77b7-4e0a-b06d-46622f97fe40-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:43.033742 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlwsg\" (UniqueName: \"kubernetes.io/projected/e672a6c1-77b7-4e0a-b06d-46622f97fe40-kube-api-access-dlwsg\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:43.033742 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033582    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-textfile\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.033742 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.033742 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e672a6c1-77b7-4e0a-b06d-46622f97fe40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:43.033742 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-wtmp\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.033742 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-accelerators-collector-config\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.034121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433b4c61-62dc-4448-b9a6-4ac3b681daf0-metrics-client-ca\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.034121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433b4c61-62dc-4448-b9a6-4ac3b681daf0-sys\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.034121 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:43.033797    2583 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 23:12:43.034121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-tls\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.034121 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:12:43.033901    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e672a6c1-77b7-4e0a-b06d-46622f97fe40-openshift-state-metrics-tls podName:e672a6c1-77b7-4e0a-b06d-46622f97fe40 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:43.533875664 +0000 UTC m=+126.468321395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e672a6c1-77b7-4e0a-b06d-46622f97fe40-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-d787c" (UID: "e672a6c1-77b7-4e0a-b06d-46622f97fe40") : secret "openshift-state-metrics-tls" not found
Sep 17 23:12:43.034121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e672a6c1-77b7-4e0a-b06d-46622f97fe40-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:43.034121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.033955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wmjd\" (UniqueName: \"kubernetes.io/projected/433b4c61-62dc-4448-b9a6-4ac3b681daf0-kube-api-access-7wmjd\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.034592 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.034572    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e672a6c1-77b7-4e0a-b06d-46622f97fe40-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:43.036191 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.036168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e672a6c1-77b7-4e0a-b06d-46622f97fe40-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:43.043596 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.043562    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlwsg\" (UniqueName: \"kubernetes.io/projected/e672a6c1-77b7-4e0a-b06d-46622f97fe40-kube-api-access-dlwsg\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:43.134902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.134811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433b4c61-62dc-4448-b9a6-4ac3b681daf0-sys\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.134902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.134862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-tls\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.134926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433b4c61-62dc-4448-b9a6-4ac3b681daf0-sys\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.134975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wmjd\" (UniqueName: \"kubernetes.io/projected/433b4c61-62dc-4448-b9a6-4ac3b681daf0-kube-api-access-7wmjd\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/433b4c61-62dc-4448-b9a6-4ac3b681daf0-root\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/433b4c61-62dc-4448-b9a6-4ac3b681daf0-root\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-textfile\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135298 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135298 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-wtmp\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135298 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-accelerators-collector-config\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135298 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433b4c61-62dc-4448-b9a6-4ac3b681daf0-metrics-client-ca\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135460 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135371    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-textfile\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135460 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135378    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-wtmp\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135785 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433b4c61-62dc-4448-b9a6-4ac3b681daf0-metrics-client-ca\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.135829 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.135814    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-accelerators-collector-config\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.137461 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.137442    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-tls\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.137555 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.137478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433b4c61-62dc-4448-b9a6-4ac3b681daf0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.148640 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.148607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wmjd\" (UniqueName: \"kubernetes.io/projected/433b4c61-62dc-4448-b9a6-4ac3b681daf0-kube-api-access-7wmjd\") pod \"node-exporter-4tmfn\" (UID: \"433b4c61-62dc-4448-b9a6-4ac3b681daf0\") " pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.290367 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.290327    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4tmfn"
Sep 17 23:12:43.302780 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:43.302751    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433b4c61_62dc_4448_b9a6_4ac3b681daf0.slice/crio-f73f9f273e68b633de114873cac89851158cb13c49047fcdaceac65d582c5ca2 WatchSource:0}: Error finding container f73f9f273e68b633de114873cac89851158cb13c49047fcdaceac65d582c5ca2: Status 404 returned error can't find the container with id f73f9f273e68b633de114873cac89851158cb13c49047fcdaceac65d582c5ca2
Sep 17 23:12:43.539904 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.539869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e672a6c1-77b7-4e0a-b06d-46622f97fe40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:43.542424 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.542389    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e672a6c1-77b7-4e0a-b06d-46622f97fe40-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-d787c\" (UID: \"e672a6c1-77b7-4e0a-b06d-46622f97fe40\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:43.690869 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.690832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4tmfn" event={"ID":"433b4c61-62dc-4448-b9a6-4ac3b681daf0","Type":"ContainerStarted","Data":"f73f9f273e68b633de114873cac89851158cb13c49047fcdaceac65d582c5ca2"}
Sep 17 23:12:43.779867 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:43.779824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"
Sep 17 23:12:44.133380 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:44.133288    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c"]
Sep 17 23:12:44.140995 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:44.140954    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode672a6c1_77b7_4e0a_b06d_46622f97fe40.slice/crio-df2481bb2773a76c910aa84a46b591eb893f570d5ea38181c6cba3aa267b12bc WatchSource:0}: Error finding container df2481bb2773a76c910aa84a46b591eb893f570d5ea38181c6cba3aa267b12bc: Status 404 returned error can't find the container with id df2481bb2773a76c910aa84a46b591eb893f570d5ea38181c6cba3aa267b12bc
Sep 17 23:12:44.696500 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:44.696405    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c" event={"ID":"e672a6c1-77b7-4e0a-b06d-46622f97fe40","Type":"ContainerStarted","Data":"320ee2ff0ea9b70d036fa6ad143cc668281c2d60da9347af354eee84e999bd23"}
Sep 17 23:12:44.696500 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:44.696474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c" event={"ID":"e672a6c1-77b7-4e0a-b06d-46622f97fe40","Type":"ContainerStarted","Data":"a4e01ea394bc88f2b8d86bbc496b7c7bfc0387682fc6b7e74a32ae0d76eb4936"}
Sep 17 23:12:44.696746 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:44.696502    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c" event={"ID":"e672a6c1-77b7-4e0a-b06d-46622f97fe40","Type":"ContainerStarted","Data":"df2481bb2773a76c910aa84a46b591eb893f570d5ea38181c6cba3aa267b12bc"}
Sep 17 23:12:44.698220 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:44.698192    2583 generic.go:356] "Generic (PLEG): container finished" podID="433b4c61-62dc-4448-b9a6-4ac3b681daf0" containerID="e07e3d23144124ecba823c4db65de28b8b8bb06007584d076b9bf2b544858657" exitCode=0
Sep 17 23:12:44.698328 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:44.698249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4tmfn" event={"ID":"433b4c61-62dc-4448-b9a6-4ac3b681daf0","Type":"ContainerDied","Data":"e07e3d23144124ecba823c4db65de28b8b8bb06007584d076b9bf2b544858657"}
Sep 17 23:12:45.704704 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:45.704607    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c" event={"ID":"e672a6c1-77b7-4e0a-b06d-46622f97fe40","Type":"ContainerStarted","Data":"6a4f4ec2d9a0d99bef07f4073bc476988fb74fb6eb9ca9ac8351c5d9c2d53cd3"}
Sep 17 23:12:45.706744 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:45.706718    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4tmfn" event={"ID":"433b4c61-62dc-4448-b9a6-4ac3b681daf0","Type":"ContainerStarted","Data":"bbaceb6a60a01de1d3af1bdbf9d24bf6d96c2cac5fa27e338f5a886fe436557e"}
Sep 17 23:12:45.706856 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:45.706766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4tmfn" event={"ID":"433b4c61-62dc-4448-b9a6-4ac3b681daf0","Type":"ContainerStarted","Data":"67c0b29b95bbdee12000ff83357e47bbfa39a90da02ee7cf582fdbc11557b6c9"}
Sep 17 23:12:45.738070 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:45.737994    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-d787c" podStartSLOduration=2.8875519499999998 podStartE2EDuration="3.737976933s" totalImagesPullingTime="850.424983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:45.736932595 +0000 UTC m=+128.671378356" watchObservedRunningTime="2026-09-17 23:12:45.737976933 +0000 UTC m=+128.672422683"
Sep 17 23:12:45.794762 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:45.794695    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4tmfn" podStartSLOduration=2.5995882999999997 podStartE2EDuration="3.794676443s" totalImagesPullingTime="1.124739185s" totalInitContainerRuntime="70.348958ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:45.793396226 +0000 UTC m=+128.727841976" watchObservedRunningTime="2026-09-17 23:12:45.794676443 +0000 UTC m=+128.729122193"
Sep 17 23:12:47.360197 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.358820    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"]
Sep 17 23:12:47.362678 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.362644    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:47.366508 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.366481    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-drvs7\""
Sep 17 23:12:47.367585 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.367560    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 23:12:47.381212 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.381183    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"]
Sep 17 23:12:47.477170 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.477129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4b4\" (UniqueName: \"kubernetes.io/projected/0fc97855-954d-4778-b07e-b85cbdaa4868-kube-api-access-5d4b4\") pod \"monitoring-plugin-5984459ff8-6jkk4\" (UID: \"0fc97855-954d-4778-b07e-b85cbdaa4868\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:47.477368 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.477207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0fc97855-954d-4778-b07e-b85cbdaa4868-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-6jkk4\" (UID: \"0fc97855-954d-4778-b07e-b85cbdaa4868\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:47.578604 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.578559    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0fc97855-954d-4778-b07e-b85cbdaa4868-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-6jkk4\" (UID: \"0fc97855-954d-4778-b07e-b85cbdaa4868\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:47.578767 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.578635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4b4\" (UniqueName: \"kubernetes.io/projected/0fc97855-954d-4778-b07e-b85cbdaa4868-kube-api-access-5d4b4\") pod \"monitoring-plugin-5984459ff8-6jkk4\" (UID: \"0fc97855-954d-4778-b07e-b85cbdaa4868\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:47.581061 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.581039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0fc97855-954d-4778-b07e-b85cbdaa4868-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-6jkk4\" (UID: \"0fc97855-954d-4778-b07e-b85cbdaa4868\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:47.599958 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.599920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4b4\" (UniqueName: \"kubernetes.io/projected/0fc97855-954d-4778-b07e-b85cbdaa4868-kube-api-access-5d4b4\") pod \"monitoring-plugin-5984459ff8-6jkk4\" (UID: \"0fc97855-954d-4778-b07e-b85cbdaa4868\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:47.673252 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.673163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:47.881384 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.881340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:12:47.883797 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.883759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73226c0-ffce-43af-89c5-e7197f34364e-metrics-certs\") pod \"network-metrics-daemon-pk7p2\" (UID: \"c73226c0-ffce-43af-89c5-e7197f34364e\") " pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:12:47.950891 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.950676    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"]
Sep 17 23:12:47.953331 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:47.953302    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc97855_954d_4778_b07e_b85cbdaa4868.slice/crio-4942c84b623a5d69a64fe4f1e69b4ed7a39f5f83ca6a81b0a77103ce0a3c7447 WatchSource:0}: Error finding container 4942c84b623a5d69a64fe4f1e69b4ed7a39f5f83ca6a81b0a77103ce0a3c7447: Status 404 returned error can't find the container with id 4942c84b623a5d69a64fe4f1e69b4ed7a39f5f83ca6a81b0a77103ce0a3c7447
Sep 17 23:12:47.978174 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.978147    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6c57\""
Sep 17 23:12:47.986443 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:47.986412    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pk7p2"
Sep 17 23:12:48.275830 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:48.275797    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pk7p2"]
Sep 17 23:12:48.278912 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:12:48.278887    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73226c0_ffce_43af_89c5_e7197f34364e.slice/crio-a55db20c6296a6292ccbfa5602f2e0149226a9ab46f4f5f2c4d4f0b9200e25fa WatchSource:0}: Error finding container a55db20c6296a6292ccbfa5602f2e0149226a9ab46f4f5f2c4d4f0b9200e25fa: Status 404 returned error can't find the container with id a55db20c6296a6292ccbfa5602f2e0149226a9ab46f4f5f2c4d4f0b9200e25fa
Sep 17 23:12:48.719517 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:48.719427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pk7p2" event={"ID":"c73226c0-ffce-43af-89c5-e7197f34364e","Type":"ContainerStarted","Data":"a55db20c6296a6292ccbfa5602f2e0149226a9ab46f4f5f2c4d4f0b9200e25fa"}
Sep 17 23:12:48.720543 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:48.720521    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4" event={"ID":"0fc97855-954d-4778-b07e-b85cbdaa4868","Type":"ContainerStarted","Data":"4942c84b623a5d69a64fe4f1e69b4ed7a39f5f83ca6a81b0a77103ce0a3c7447"}
Sep 17 23:12:49.727544 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:49.727458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pk7p2" event={"ID":"c73226c0-ffce-43af-89c5-e7197f34364e","Type":"ContainerStarted","Data":"a7c4908e6d3e17c6db1f2bb2d2282cc4c687966c6017540822a0664b7f855c7c"}
Sep 17 23:12:49.728959 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:49.728401    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pk7p2" event={"ID":"c73226c0-ffce-43af-89c5-e7197f34364e","Type":"ContainerStarted","Data":"b098924a2188209ab9308e32c100915c2f70233d5014b6a0d9cb189b968a3802"}
Sep 17 23:12:49.730791 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:49.730741    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4" event={"ID":"0fc97855-954d-4778-b07e-b85cbdaa4868","Type":"ContainerStarted","Data":"f6d74a872b4abdc9142dd9e64dfaa6faeb7686f41979270c637ea534d76321e5"}
Sep 17 23:12:49.731000 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:49.730961    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:49.736216 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:49.736164    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4"
Sep 17 23:12:49.761737 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:49.761686    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pk7p2" podStartSLOduration=129.553977496 podStartE2EDuration="2m10.761672522s" totalImagesPullingTime="1.207695026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:49.761368844 +0000 UTC m=+132.695814595" watchObservedRunningTime="2026-09-17 23:12:49.761672522 +0000 UTC m=+132.696118272"
Sep 17 23:12:49.813131 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:49.813076    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-6jkk4" podStartSLOduration=1.28198862 podStartE2EDuration="2.813058743s" totalImagesPullingTime="1.531070123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:49.811769332 +0000 UTC m=+132.746215064" watchObservedRunningTime="2026-09-17 23:12:49.813058743 +0000 UTC m=+132.747504492"
Sep 17 23:12:54.151826 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:54.151790    2583 scope.go:117] "RemoveContainer" containerID="3e19e47fc2858990447376a02ea0fafbab4dfa71e10460c88a141c83f4ec8260"
Sep 17 23:12:54.752009 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:54.751981    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:12:54.752203 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:54.752047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" event={"ID":"a31ae514-8497-4e88-89e7-fa894e336f26","Type":"ContainerStarted","Data":"01afd0ccf7bdb02859f55d54b0c6fbc98909f46706ac8043156ba9d4a77c99a5"}
Sep 17 23:12:54.752346 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:54.752318    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:12:54.757297 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:54.757272    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q"
Sep 17 23:12:54.772823 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:54.772763    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-g5c6q" podStartSLOduration=55.021700953 podStartE2EDuration="56.772734908s" totalImagesPullingTime="1.751033955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:54.771534614 +0000 UTC m=+137.705980378" watchObservedRunningTime="2026-09-17 23:12:54.772734908 +0000 UTC m=+137.707180658"
Sep 17 23:12:57.588240 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:12:57.588198    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74dcc75879-rt6s8"]
Sep 17 23:13:18.840779 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:18.840742    2583 generic.go:356] "Generic (PLEG): container finished" podID="d1c623b6-ba24-484f-bdc0-06dd1a93f1b2" containerID="ce1f626d8e56f570bc89cb12ca347c22f904f2012160d414bef37c5e2489d502" exitCode=0
Sep 17 23:13:18.841319 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:18.840826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf" event={"ID":"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2","Type":"ContainerDied","Data":"ce1f626d8e56f570bc89cb12ca347c22f904f2012160d414bef37c5e2489d502"}
Sep 17 23:13:18.841319 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:18.841304    2583 scope.go:117] "RemoveContainer" containerID="ce1f626d8e56f570bc89cb12ca347c22f904f2012160d414bef37c5e2489d502"
Sep 17 23:13:19.846672 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:19.846642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6hmnf" event={"ID":"d1c623b6-ba24-484f-bdc0-06dd1a93f1b2","Type":"ContainerStarted","Data":"7905dd1248c9f99ad63192cfb8e6a95a6f6706370297376ff2dac4830d3ffad7"}
Sep 17 23:13:22.607566 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.607505    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8" podUID="09eaea72-e75d-4676-8fc0-e528b568a81f" containerName="registry" containerID="cri-o://28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e" gracePeriod=30
Sep 17 23:13:22.849654 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.849631    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:13:22.858796 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.858741    2583 generic.go:356] "Generic (PLEG): container finished" podID="09eaea72-e75d-4676-8fc0-e528b568a81f" containerID="28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e" exitCode=0
Sep 17 23:13:22.858796 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.858782    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8"
Sep 17 23:13:22.858954 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.858815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8" event={"ID":"09eaea72-e75d-4676-8fc0-e528b568a81f","Type":"ContainerDied","Data":"28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e"}
Sep 17 23:13:22.858954 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.858882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74dcc75879-rt6s8" event={"ID":"09eaea72-e75d-4676-8fc0-e528b568a81f","Type":"ContainerDied","Data":"543c872a7dbe01e0525a25a2f6653d457b3b1888fe97c967777caff21c040cad"}
Sep 17 23:13:22.858954 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.858904    2583 scope.go:117] "RemoveContainer" containerID="28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e"
Sep 17 23:13:22.867932 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.867915    2583 scope.go:117] "RemoveContainer" containerID="28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e"
Sep 17 23:13:22.868263 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:13:22.868241    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e\": container with ID starting with 28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e not found: ID does not exist" containerID="28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e"
Sep 17 23:13:22.868360 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.868269    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e"} err="failed to get container status \"28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e\": rpc error: code = NotFound desc = could not find container \"28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e\": container with ID starting with 28cd0c871deba429bfdb66a38142625463374563ff942d799feb68c3654fed7e not found: ID does not exist"
Sep 17 23:13:22.960637 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.960608    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-installation-pull-secrets\") pod \"09eaea72-e75d-4676-8fc0-e528b568a81f\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") "
Sep 17 23:13:22.960637 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.960641    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-certificates\") pod \"09eaea72-e75d-4676-8fc0-e528b568a81f\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") "
Sep 17 23:13:22.960884 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.960664    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09eaea72-e75d-4676-8fc0-e528b568a81f-ca-trust-extracted\") pod \"09eaea72-e75d-4676-8fc0-e528b568a81f\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") "
Sep 17 23:13:22.960884 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.960702    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-image-registry-private-configuration\") pod \"09eaea72-e75d-4676-8fc0-e528b568a81f\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") "
Sep 17 23:13:22.960884 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.960729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw9n9\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-kube-api-access-nw9n9\") pod \"09eaea72-e75d-4676-8fc0-e528b568a81f\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") "
Sep 17 23:13:22.960884 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.960770    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-bound-sa-token\") pod \"09eaea72-e75d-4676-8fc0-e528b568a81f\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") "
Sep 17 23:13:22.961119 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.960941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") pod \"09eaea72-e75d-4676-8fc0-e528b568a81f\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") "
Sep 17 23:13:22.961119 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.960990    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-trusted-ca\") pod \"09eaea72-e75d-4676-8fc0-e528b568a81f\" (UID: \"09eaea72-e75d-4676-8fc0-e528b568a81f\") "
Sep 17 23:13:22.961119 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.961084    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "09eaea72-e75d-4676-8fc0-e528b568a81f" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:22.961460 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.961425    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-certificates\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:13:22.961586 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.961479    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "09eaea72-e75d-4676-8fc0-e528b568a81f" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:22.963513 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.963469    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "09eaea72-e75d-4676-8fc0-e528b568a81f" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:22.963513 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.963499    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-kube-api-access-nw9n9" (OuterVolumeSpecName: "kube-api-access-nw9n9") pod "09eaea72-e75d-4676-8fc0-e528b568a81f" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f"). InnerVolumeSpecName "kube-api-access-nw9n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:22.963673 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.963556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "09eaea72-e75d-4676-8fc0-e528b568a81f" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:22.963772 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.963750    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "09eaea72-e75d-4676-8fc0-e528b568a81f" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:22.963812 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.963755    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "09eaea72-e75d-4676-8fc0-e528b568a81f" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:22.969592 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:22.969571    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09eaea72-e75d-4676-8fc0-e528b568a81f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "09eaea72-e75d-4676-8fc0-e528b568a81f" (UID: "09eaea72-e75d-4676-8fc0-e528b568a81f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:13:23.062480 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:23.062437    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-image-registry-private-configuration\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:13:23.062480 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:23.062480    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9n9\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-kube-api-access-nw9n9\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:13:23.062669 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:23.062498    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-bound-sa-token\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:13:23.062669 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:23.062512    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09eaea72-e75d-4676-8fc0-e528b568a81f-registry-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:13:23.062669 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:23.062524    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09eaea72-e75d-4676-8fc0-e528b568a81f-trusted-ca\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:13:23.062669 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:23.062536    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09eaea72-e75d-4676-8fc0-e528b568a81f-installation-pull-secrets\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:13:23.062669 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:23.062550    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09eaea72-e75d-4676-8fc0-e528b568a81f-ca-trust-extracted\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:13:23.180257 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:23.180225    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74dcc75879-rt6s8"]
Sep 17 23:13:23.185263 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:23.185239    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74dcc75879-rt6s8"]
Sep 17 23:13:25.155957 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:13:25.155920    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09eaea72-e75d-4676-8fc0-e528b568a81f" path="/var/lib/kubelet/pods/09eaea72-e75d-4676-8fc0-e528b568a81f/volumes"
Sep 17 23:15:39.042375 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:15:39.042348    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:15:39.045495 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:15:39.045471    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:15:39.049498 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:15:39.049476    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:15:39.053324 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:15:39.053302    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:15:39.053480 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:15:39.053377    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:16:40.150779 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.150744    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"]
Sep 17 23:16:40.151353 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.151159    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09eaea72-e75d-4676-8fc0-e528b568a81f" containerName="registry"
Sep 17 23:16:40.151353 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.151178    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09eaea72-e75d-4676-8fc0-e528b568a81f" containerName="registry"
Sep 17 23:16:40.151353 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.151264    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09eaea72-e75d-4676-8fc0-e528b568a81f" containerName="registry"
Sep 17 23:16:40.154345 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.154319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.158288 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.158270    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 23:16:40.158617 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.158601    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s7gpl\""
Sep 17 23:16:40.158936 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.158919    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 23:16:40.168780 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.168745    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"]
Sep 17 23:16:40.222468 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.222427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.222650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.222474    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcgnz\" (UniqueName: \"kubernetes.io/projected/ef81cd30-f178-4ea3-b62b-badc06f9088c-kube-api-access-lcgnz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.222650 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.222553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.323119 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.323064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.323119 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.323121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcgnz\" (UniqueName: \"kubernetes.io/projected/ef81cd30-f178-4ea3-b62b-badc06f9088c-kube-api-access-lcgnz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.323371 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.323157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.323453 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.323431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.323507 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.323462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.333658 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.333622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcgnz\" (UniqueName: \"kubernetes.io/projected/ef81cd30-f178-4ea3-b62b-badc06f9088c-kube-api-access-lcgnz\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.464210 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.464103    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:40.738975 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.738946    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"]
Sep 17 23:16:40.741531 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:16:40.741497    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef81cd30_f178_4ea3_b62b_badc06f9088c.slice/crio-5601c6f8d707fb450f31d274de1d6bf91aa9b595ed337d3988f01777aa6d96c9 WatchSource:0}: Error finding container 5601c6f8d707fb450f31d274de1d6bf91aa9b595ed337d3988f01777aa6d96c9: Status 404 returned error can't find the container with id 5601c6f8d707fb450f31d274de1d6bf91aa9b595ed337d3988f01777aa6d96c9
Sep 17 23:16:40.743394 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:40.743377    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:16:41.582580 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:41.582540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc" event={"ID":"ef81cd30-f178-4ea3-b62b-badc06f9088c","Type":"ContainerStarted","Data":"5601c6f8d707fb450f31d274de1d6bf91aa9b595ed337d3988f01777aa6d96c9"}
Sep 17 23:16:47.610500 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:47.610465    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerID="03d329cee0f0db56e07c84dea9fe01b8a8983f187fe499c701c49f4346f4326f" exitCode=0
Sep 17 23:16:47.610900 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:47.610553    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc" event={"ID":"ef81cd30-f178-4ea3-b62b-badc06f9088c","Type":"ContainerDied","Data":"03d329cee0f0db56e07c84dea9fe01b8a8983f187fe499c701c49f4346f4326f"}
Sep 17 23:16:50.624193 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:50.624153    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerID="c6119c70edfbf08c77dc104cda632977c296f67d7ce0cda946f2bd3993726134" exitCode=0
Sep 17 23:16:50.624649 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:50.624244    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc" event={"ID":"ef81cd30-f178-4ea3-b62b-badc06f9088c","Type":"ContainerDied","Data":"c6119c70edfbf08c77dc104cda632977c296f67d7ce0cda946f2bd3993726134"}
Sep 17 23:16:57.656426 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:57.656391    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerID="ce53caa8d59699137ee638e19346a8ab57023eefb6f863b5a993b9eab0a45fab" exitCode=0
Sep 17 23:16:57.656828 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:57.656465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc" event={"ID":"ef81cd30-f178-4ea3-b62b-badc06f9088c","Type":"ContainerDied","Data":"ce53caa8d59699137ee638e19346a8ab57023eefb6f863b5a993b9eab0a45fab"}
Sep 17 23:16:58.779710 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.779684    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:58.855253 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.855214    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-util\") pod \"ef81cd30-f178-4ea3-b62b-badc06f9088c\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") "
Sep 17 23:16:58.855461 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.855273    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-bundle\") pod \"ef81cd30-f178-4ea3-b62b-badc06f9088c\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") "
Sep 17 23:16:58.855461 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.855312    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcgnz\" (UniqueName: \"kubernetes.io/projected/ef81cd30-f178-4ea3-b62b-badc06f9088c-kube-api-access-lcgnz\") pod \"ef81cd30-f178-4ea3-b62b-badc06f9088c\" (UID: \"ef81cd30-f178-4ea3-b62b-badc06f9088c\") "
Sep 17 23:16:58.855942 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.855908    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-bundle" (OuterVolumeSpecName: "bundle") pod "ef81cd30-f178-4ea3-b62b-badc06f9088c" (UID: "ef81cd30-f178-4ea3-b62b-badc06f9088c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:16:58.857667 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.857643    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef81cd30-f178-4ea3-b62b-badc06f9088c-kube-api-access-lcgnz" (OuterVolumeSpecName: "kube-api-access-lcgnz") pod "ef81cd30-f178-4ea3-b62b-badc06f9088c" (UID: "ef81cd30-f178-4ea3-b62b-badc06f9088c"). InnerVolumeSpecName "kube-api-access-lcgnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:16:58.860273 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.860249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-util" (OuterVolumeSpecName: "util") pod "ef81cd30-f178-4ea3-b62b-badc06f9088c" (UID: "ef81cd30-f178-4ea3-b62b-badc06f9088c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:16:58.956492 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.956409    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:16:58.956492 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.956439    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcgnz\" (UniqueName: \"kubernetes.io/projected/ef81cd30-f178-4ea3-b62b-badc06f9088c-kube-api-access-lcgnz\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:16:58.956492 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:58.956450    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef81cd30-f178-4ea3-b62b-badc06f9088c-util\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:16:59.665833 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:59.665798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc" event={"ID":"ef81cd30-f178-4ea3-b62b-badc06f9088c","Type":"ContainerDied","Data":"5601c6f8d707fb450f31d274de1d6bf91aa9b595ed337d3988f01777aa6d96c9"}
Sep 17 23:16:59.665833 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:59.665833    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1arxsqc"
Sep 17 23:16:59.666060 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:16:59.665846    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5601c6f8d707fb450f31d274de1d6bf91aa9b595ed337d3988f01777aa6d96c9"
Sep 17 23:17:04.093469 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.093436    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"]
Sep 17 23:17:04.093859 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.093706    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerName="util"
Sep 17 23:17:04.093859 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.093717    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerName="util"
Sep 17 23:17:04.093859 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.093728    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerName="extract"
Sep 17 23:17:04.093859 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.093733    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerName="extract"
Sep 17 23:17:04.093859 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.093739    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerName="pull"
Sep 17 23:17:04.093859 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.093745    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerName="pull"
Sep 17 23:17:04.093859 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.093804    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef81cd30-f178-4ea3-b62b-badc06f9088c" containerName="extract"
Sep 17 23:17:04.134197 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.134153    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"]
Sep 17 23:17:04.134357 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.134211    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:04.137927 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.137900    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-zrp4d\""
Sep 17 23:17:04.137927 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.137922    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 23:17:04.138588 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.138572    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 23:17:04.139091 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.139075    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 23:17:04.195874 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.195834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64z62\" (UniqueName: \"kubernetes.io/projected/043c8751-90c9-4bd0-9976-e7d5fe4fb3bc-kube-api-access-64z62\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bvtsx\" (UID: \"043c8751-90c9-4bd0-9976-e7d5fe4fb3bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:04.195874 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.195869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/043c8751-90c9-4bd0-9976-e7d5fe4fb3bc-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bvtsx\" (UID: \"043c8751-90c9-4bd0-9976-e7d5fe4fb3bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:04.297333 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.297293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64z62\" (UniqueName: \"kubernetes.io/projected/043c8751-90c9-4bd0-9976-e7d5fe4fb3bc-kube-api-access-64z62\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bvtsx\" (UID: \"043c8751-90c9-4bd0-9976-e7d5fe4fb3bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:04.297333 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.297334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/043c8751-90c9-4bd0-9976-e7d5fe4fb3bc-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bvtsx\" (UID: \"043c8751-90c9-4bd0-9976-e7d5fe4fb3bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:04.299683 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.299650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/043c8751-90c9-4bd0-9976-e7d5fe4fb3bc-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bvtsx\" (UID: \"043c8751-90c9-4bd0-9976-e7d5fe4fb3bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:04.319216 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.319183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64z62\" (UniqueName: \"kubernetes.io/projected/043c8751-90c9-4bd0-9976-e7d5fe4fb3bc-kube-api-access-64z62\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-bvtsx\" (UID: \"043c8751-90c9-4bd0-9976-e7d5fe4fb3bc\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:04.444228 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.444143    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:04.742293 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:04.742268    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"]
Sep 17 23:17:04.744156 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:17:04.744127    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043c8751_90c9_4bd0_9976_e7d5fe4fb3bc.slice/crio-0eb772326393ee98690b3d4f1a8a40f845a1ad526cab4f389682dec64d13fb22 WatchSource:0}: Error finding container 0eb772326393ee98690b3d4f1a8a40f845a1ad526cab4f389682dec64d13fb22: Status 404 returned error can't find the container with id 0eb772326393ee98690b3d4f1a8a40f845a1ad526cab4f389682dec64d13fb22
Sep 17 23:17:05.690375 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:05.690321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx" event={"ID":"043c8751-90c9-4bd0-9976-e7d5fe4fb3bc","Type":"ContainerStarted","Data":"0eb772326393ee98690b3d4f1a8a40f845a1ad526cab4f389682dec64d13fb22"}
Sep 17 23:17:08.703652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.703604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx" event={"ID":"043c8751-90c9-4bd0-9976-e7d5fe4fb3bc","Type":"ContainerStarted","Data":"9ecd73a0f2c321ce8330a756c3d624fc5adfea3b67d324f23e7bc4842bfdf52a"}
Sep 17 23:17:08.704153 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.703744    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:08.732607 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.732550    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx" podStartSLOduration=1.7305624480000001 podStartE2EDuration="4.732533438s" totalImagesPullingTime="3.00197099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:08.731701183 +0000 UTC m=+391.666146943" watchObservedRunningTime="2026-09-17 23:17:08.732533438 +0000 UTC m=+391.666979187"
Sep 17 23:17:08.781615 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.781579    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-4dzzv"]
Sep 17 23:17:08.784812 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.784793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:08.787832 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.787813    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 23:17:08.790477 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.790461    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-wcgbk\""
Sep 17 23:17:08.792415 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.792397    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 23:17:08.806730 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.806703    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-4dzzv"]
Sep 17 23:17:08.835294 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.835267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-cabundle0\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:08.835440 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.835308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl8vd\" (UniqueName: \"kubernetes.io/projected/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-kube-api-access-gl8vd\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:08.835440 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.835338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-certificates\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:08.936634 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.936599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-cabundle0\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:08.936857 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.936647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8vd\" (UniqueName: \"kubernetes.io/projected/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-kube-api-access-gl8vd\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:08.936857 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.936777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-certificates\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:08.936984 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:17:08.936901    2583 secret.go:281] references non-existent secret key: ca.crt
Sep 17 23:17:08.936984 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:17:08.936921    2583 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 23:17:08.936984 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:17:08.936932    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-4dzzv: references non-existent secret key: ca.crt
Sep 17 23:17:08.937177 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:17:08.936995    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-certificates podName:4d7f5fdb-e754-4290-ba1d-af48ac1a331a nodeName:}" failed. No retries permitted until 2026-09-17 23:17:09.436975997 +0000 UTC m=+392.371421738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-certificates") pod "keda-operator-579c8958f7-4dzzv" (UID: "4d7f5fdb-e754-4290-ba1d-af48ac1a331a") : references non-existent secret key: ca.crt
Sep 17 23:17:08.937438 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.937414    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-cabundle0\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:08.967532 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:08.967466    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8vd\" (UniqueName: \"kubernetes.io/projected/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-kube-api-access-gl8vd\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:09.365664 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.365636    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"]
Sep 17 23:17:09.368969 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.368953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.375162 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.375137    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 23:17:09.394860 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.394830    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"]
Sep 17 23:17:09.441929 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.441894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-certificates\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:09.441929 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.441931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0e168316-e789-4558-ad47-e84eb06ab489-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-77bvw\" (UID: \"0e168316-e789-4558-ad47-e84eb06ab489\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.442184 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.441965    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0e168316-e789-4558-ad47-e84eb06ab489-certificates\") pod \"keda-metrics-apiserver-77857b7b98-77bvw\" (UID: \"0e168316-e789-4558-ad47-e84eb06ab489\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.442184 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.442008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjpq\" (UniqueName: \"kubernetes.io/projected/0e168316-e789-4558-ad47-e84eb06ab489-kube-api-access-vwjpq\") pod \"keda-metrics-apiserver-77857b7b98-77bvw\" (UID: \"0e168316-e789-4558-ad47-e84eb06ab489\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.444787 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.444749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4d7f5fdb-e754-4290-ba1d-af48ac1a331a-certificates\") pod \"keda-operator-579c8958f7-4dzzv\" (UID: \"4d7f5fdb-e754-4290-ba1d-af48ac1a331a\") " pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:09.543216 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.543183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjpq\" (UniqueName: \"kubernetes.io/projected/0e168316-e789-4558-ad47-e84eb06ab489-kube-api-access-vwjpq\") pod \"keda-metrics-apiserver-77857b7b98-77bvw\" (UID: \"0e168316-e789-4558-ad47-e84eb06ab489\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.543403 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.543290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0e168316-e789-4558-ad47-e84eb06ab489-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-77bvw\" (UID: \"0e168316-e789-4558-ad47-e84eb06ab489\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.543403 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.543324    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0e168316-e789-4558-ad47-e84eb06ab489-certificates\") pod \"keda-metrics-apiserver-77857b7b98-77bvw\" (UID: \"0e168316-e789-4558-ad47-e84eb06ab489\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.543676 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.543654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0e168316-e789-4558-ad47-e84eb06ab489-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-77bvw\" (UID: \"0e168316-e789-4558-ad47-e84eb06ab489\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.545833 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.545814    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0e168316-e789-4558-ad47-e84eb06ab489-certificates\") pod \"keda-metrics-apiserver-77857b7b98-77bvw\" (UID: \"0e168316-e789-4558-ad47-e84eb06ab489\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.570842 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.570812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjpq\" (UniqueName: \"kubernetes.io/projected/0e168316-e789-4558-ad47-e84eb06ab489-kube-api-access-vwjpq\") pod \"keda-metrics-apiserver-77857b7b98-77bvw\" (UID: \"0e168316-e789-4558-ad47-e84eb06ab489\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.661655 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.661556    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ctbcc"]
Sep 17 23:17:09.665264 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.665243    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:17:09.673936 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.673915    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 23:17:09.679489 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.679467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:09.685280 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.685254    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ctbcc"]
Sep 17 23:17:09.694467 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.694429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:09.745521 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.745472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/132ff0a7-6db6-460d-9b8e-d2049f2c4a43-certificates\") pod \"keda-admission-6c4d864b86-ctbcc\" (UID: \"132ff0a7-6db6-460d-9b8e-d2049f2c4a43\") " pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:17:09.746012 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.745606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr487\" (UniqueName: \"kubernetes.io/projected/132ff0a7-6db6-460d-9b8e-d2049f2c4a43-kube-api-access-zr487\") pod \"keda-admission-6c4d864b86-ctbcc\" (UID: \"132ff0a7-6db6-460d-9b8e-d2049f2c4a43\") " pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:17:09.846819 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.846787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr487\" (UniqueName: \"kubernetes.io/projected/132ff0a7-6db6-460d-9b8e-d2049f2c4a43-kube-api-access-zr487\") pod \"keda-admission-6c4d864b86-ctbcc\" (UID: \"132ff0a7-6db6-460d-9b8e-d2049f2c4a43\") " pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:17:09.846999 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.846835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/132ff0a7-6db6-460d-9b8e-d2049f2c4a43-certificates\") pod \"keda-admission-6c4d864b86-ctbcc\" (UID: \"132ff0a7-6db6-460d-9b8e-d2049f2c4a43\") " pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:17:09.849432 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.849406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/132ff0a7-6db6-460d-9b8e-d2049f2c4a43-certificates\") pod \"keda-admission-6c4d864b86-ctbcc\" (UID: \"132ff0a7-6db6-460d-9b8e-d2049f2c4a43\") " pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:17:09.860671 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.860650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr487\" (UniqueName: \"kubernetes.io/projected/132ff0a7-6db6-460d-9b8e-d2049f2c4a43-kube-api-access-zr487\") pod \"keda-admission-6c4d864b86-ctbcc\" (UID: \"132ff0a7-6db6-460d-9b8e-d2049f2c4a43\") " pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:17:09.977277 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.976948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:17:09.982339 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:09.980844    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"]
Sep 17 23:17:10.003209 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:10.003172    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-4dzzv"]
Sep 17 23:17:10.009525 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:17:10.009493    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7f5fdb_e754_4290_ba1d_af48ac1a331a.slice/crio-b77db0b993e06fe8b5a54e8b21f96dd162e6da0eda39ef4b23d05cc7cd2653d9 WatchSource:0}: Error finding container b77db0b993e06fe8b5a54e8b21f96dd162e6da0eda39ef4b23d05cc7cd2653d9: Status 404 returned error can't find the container with id b77db0b993e06fe8b5a54e8b21f96dd162e6da0eda39ef4b23d05cc7cd2653d9
Sep 17 23:17:10.283744 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:10.283711    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ctbcc"]
Sep 17 23:17:10.287105 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:17:10.287078    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132ff0a7_6db6_460d_9b8e_d2049f2c4a43.slice/crio-f9a42d1bb1cd1b55d09f711651eef289e6018dcac385099330dfb4b1edfed295 WatchSource:0}: Error finding container f9a42d1bb1cd1b55d09f711651eef289e6018dcac385099330dfb4b1edfed295: Status 404 returned error can't find the container with id f9a42d1bb1cd1b55d09f711651eef289e6018dcac385099330dfb4b1edfed295
Sep 17 23:17:10.717247 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:10.717157    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw" event={"ID":"0e168316-e789-4558-ad47-e84eb06ab489","Type":"ContainerStarted","Data":"778cbf25d495b48b72af1fa4ca232ca5bdbc8252e403c22e1dc6419b9ba16632"}
Sep 17 23:17:10.718965 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:10.718884    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-4dzzv" event={"ID":"4d7f5fdb-e754-4290-ba1d-af48ac1a331a","Type":"ContainerStarted","Data":"b77db0b993e06fe8b5a54e8b21f96dd162e6da0eda39ef4b23d05cc7cd2653d9"}
Sep 17 23:17:10.720369 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:10.720332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ctbcc" event={"ID":"132ff0a7-6db6-460d-9b8e-d2049f2c4a43","Type":"ContainerStarted","Data":"f9a42d1bb1cd1b55d09f711651eef289e6018dcac385099330dfb4b1edfed295"}
Sep 17 23:17:14.742487 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:14.742449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ctbcc" event={"ID":"132ff0a7-6db6-460d-9b8e-d2049f2c4a43","Type":"ContainerStarted","Data":"d36778ace46e0326c29eab55351677e95a595d8445b6af8e0836f8e688c505f2"}
Sep 17 23:17:14.742964 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:14.742527    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:17:14.744198 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:14.744168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw" event={"ID":"0e168316-e789-4558-ad47-e84eb06ab489","Type":"ContainerStarted","Data":"64c52bb7a0b4e2b161c9a809c40bb16df90ab427f59e3d336d456f4fb4596424"}
Sep 17 23:17:14.744327 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:14.744253    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:14.745883 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:14.745857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-4dzzv" event={"ID":"4d7f5fdb-e754-4290-ba1d-af48ac1a331a","Type":"ContainerStarted","Data":"254e01a3e1d9e19ddf74d489975b790d495e74a3aa2af4cbcd3c340a75325eb4"}
Sep 17 23:17:14.745998 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:14.745915    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:14.762652 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:14.762606    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-ctbcc" podStartSLOduration=4.133782858 podStartE2EDuration="5.762592031s" totalImagesPullingTime="1.628809173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:14.761074171 +0000 UTC m=+397.695519920" watchObservedRunningTime="2026-09-17 23:17:14.762592031 +0000 UTC m=+397.697037781"
Sep 17 23:17:14.780366 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:14.780324    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw" podStartSLOduration=1.871012763 podStartE2EDuration="5.780306878s" totalImagesPullingTime="3.909294115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:14.779581759 +0000 UTC m=+397.714027534" watchObservedRunningTime="2026-09-17 23:17:14.780306878 +0000 UTC m=+397.714752633"
Sep 17 23:17:14.798002 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:14.797958    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-4dzzv" podStartSLOduration=2.91408391 podStartE2EDuration="6.797944455s" totalImagesPullingTime="3.883860545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:17:14.796620099 +0000 UTC m=+397.731065850" watchObservedRunningTime="2026-09-17 23:17:14.797944455 +0000 UTC m=+397.732390204"
Sep 17 23:17:25.755299 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:25.755264    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77bvw"
Sep 17 23:17:29.711729 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:29.711698    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-bvtsx"
Sep 17 23:17:35.753322 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:35.753293    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-4dzzv"
Sep 17 23:17:35.753775 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:17:35.753622    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-ctbcc"
Sep 17 23:18:25.948299 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:25.948261    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5fcd7899-xs9tq"]
Sep 17 23:18:25.951643 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:25.951619    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:25.954290 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:25.954262    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:18:25.954427 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:25.954263    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 23:18:25.954427 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:25.954376    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-ljphj\""
Sep 17 23:18:25.955218 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:25.955203    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:18:25.962926 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:25.962898    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5fcd7899-xs9tq"]
Sep 17 23:18:26.072221 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:26.072180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63cb1c90-d64b-4c40-ab92-7f430eebac3e-cert\") pod \"kserve-controller-manager-5c5fcd7899-xs9tq\" (UID: \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\") " pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:26.072221 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:26.072223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghrk\" (UniqueName: \"kubernetes.io/projected/63cb1c90-d64b-4c40-ab92-7f430eebac3e-kube-api-access-tghrk\") pod \"kserve-controller-manager-5c5fcd7899-xs9tq\" (UID: \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\") " pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:26.173624 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:26.173586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63cb1c90-d64b-4c40-ab92-7f430eebac3e-cert\") pod \"kserve-controller-manager-5c5fcd7899-xs9tq\" (UID: \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\") " pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:26.173624 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:26.173626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghrk\" (UniqueName: \"kubernetes.io/projected/63cb1c90-d64b-4c40-ab92-7f430eebac3e-kube-api-access-tghrk\") pod \"kserve-controller-manager-5c5fcd7899-xs9tq\" (UID: \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\") " pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:26.173834 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:18:26.173769    2583 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 23:18:26.173870 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:18:26.173845    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63cb1c90-d64b-4c40-ab92-7f430eebac3e-cert podName:63cb1c90-d64b-4c40-ab92-7f430eebac3e nodeName:}" failed. No retries permitted until 2026-09-17 23:18:26.673824789 +0000 UTC m=+469.608270517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/63cb1c90-d64b-4c40-ab92-7f430eebac3e-cert") pod "kserve-controller-manager-5c5fcd7899-xs9tq" (UID: "63cb1c90-d64b-4c40-ab92-7f430eebac3e") : secret "kserve-webhook-server-cert" not found
Sep 17 23:18:26.184672 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:26.184646    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghrk\" (UniqueName: \"kubernetes.io/projected/63cb1c90-d64b-4c40-ab92-7f430eebac3e-kube-api-access-tghrk\") pod \"kserve-controller-manager-5c5fcd7899-xs9tq\" (UID: \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\") " pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:26.679415 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:26.679375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63cb1c90-d64b-4c40-ab92-7f430eebac3e-cert\") pod \"kserve-controller-manager-5c5fcd7899-xs9tq\" (UID: \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\") " pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:26.681879 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:26.681858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63cb1c90-d64b-4c40-ab92-7f430eebac3e-cert\") pod \"kserve-controller-manager-5c5fcd7899-xs9tq\" (UID: \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\") " pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:26.862226 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:26.862188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:27.132350 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:27.132319    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5fcd7899-xs9tq"]
Sep 17 23:18:27.135505 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:18:27.135477    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cb1c90_d64b_4c40_ab92_7f430eebac3e.slice/crio-cfc582b721175cb6b6bb111ba4fea90512b0f433d26e3320f157618423ba6926 WatchSource:0}: Error finding container cfc582b721175cb6b6bb111ba4fea90512b0f433d26e3320f157618423ba6926: Status 404 returned error can't find the container with id cfc582b721175cb6b6bb111ba4fea90512b0f433d26e3320f157618423ba6926
Sep 17 23:18:28.059388 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:28.059346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq" event={"ID":"63cb1c90-d64b-4c40-ab92-7f430eebac3e","Type":"ContainerStarted","Data":"cfc582b721175cb6b6bb111ba4fea90512b0f433d26e3320f157618423ba6926"}
Sep 17 23:18:30.074634 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:30.074588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq" event={"ID":"63cb1c90-d64b-4c40-ab92-7f430eebac3e","Type":"ContainerStarted","Data":"3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188"}
Sep 17 23:18:30.075055 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:30.074755    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:18:30.097058 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:18:30.096984    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq" podStartSLOduration=2.607031909 podStartE2EDuration="5.096966722s" totalImagesPullingTime="2.489934813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:30.094284696 +0000 UTC m=+473.028730446" watchObservedRunningTime="2026-09-17 23:18:30.096966722 +0000 UTC m=+473.031412472"
Sep 17 23:19:01.083518 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:01.083489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:19:10.160281 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.160243    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5fcd7899-xs9tq"]
Sep 17 23:19:10.160777 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.160462    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq" podUID="63cb1c90-d64b-4c40-ab92-7f430eebac3e" containerName="manager" containerID="cri-o://3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188" gracePeriod=10
Sep 17 23:19:10.184126 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.184098    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5fcd7899-kx7cf"]
Sep 17 23:19:10.191822 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.191785    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:10.197391 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.197362    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5fcd7899-kx7cf"]
Sep 17 23:19:10.324656 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.324618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95n6r\" (UniqueName: \"kubernetes.io/projected/b11faebd-ccf3-4927-909b-8cdbd4901abf-kube-api-access-95n6r\") pod \"kserve-controller-manager-5c5fcd7899-kx7cf\" (UID: \"b11faebd-ccf3-4927-909b-8cdbd4901abf\") " pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:10.324836 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.324677    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b11faebd-ccf3-4927-909b-8cdbd4901abf-cert\") pod \"kserve-controller-manager-5c5fcd7899-kx7cf\" (UID: \"b11faebd-ccf3-4927-909b-8cdbd4901abf\") " pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:10.403636 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.403612    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:19:10.425169 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.425075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95n6r\" (UniqueName: \"kubernetes.io/projected/b11faebd-ccf3-4927-909b-8cdbd4901abf-kube-api-access-95n6r\") pod \"kserve-controller-manager-5c5fcd7899-kx7cf\" (UID: \"b11faebd-ccf3-4927-909b-8cdbd4901abf\") " pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:10.425169 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.425140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b11faebd-ccf3-4927-909b-8cdbd4901abf-cert\") pod \"kserve-controller-manager-5c5fcd7899-kx7cf\" (UID: \"b11faebd-ccf3-4927-909b-8cdbd4901abf\") " pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:10.427857 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.427827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b11faebd-ccf3-4927-909b-8cdbd4901abf-cert\") pod \"kserve-controller-manager-5c5fcd7899-kx7cf\" (UID: \"b11faebd-ccf3-4927-909b-8cdbd4901abf\") " pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:10.434227 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.434197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95n6r\" (UniqueName: \"kubernetes.io/projected/b11faebd-ccf3-4927-909b-8cdbd4901abf-kube-api-access-95n6r\") pod \"kserve-controller-manager-5c5fcd7899-kx7cf\" (UID: \"b11faebd-ccf3-4927-909b-8cdbd4901abf\") " pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:10.525502 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.525470    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63cb1c90-d64b-4c40-ab92-7f430eebac3e-cert\") pod \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\" (UID: \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\") "
Sep 17 23:19:10.525683 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.525543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tghrk\" (UniqueName: \"kubernetes.io/projected/63cb1c90-d64b-4c40-ab92-7f430eebac3e-kube-api-access-tghrk\") pod \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\" (UID: \"63cb1c90-d64b-4c40-ab92-7f430eebac3e\") "
Sep 17 23:19:10.527676 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.527648    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63cb1c90-d64b-4c40-ab92-7f430eebac3e-cert" (OuterVolumeSpecName: "cert") pod "63cb1c90-d64b-4c40-ab92-7f430eebac3e" (UID: "63cb1c90-d64b-4c40-ab92-7f430eebac3e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:19:10.527770 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.527687    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cb1c90-d64b-4c40-ab92-7f430eebac3e-kube-api-access-tghrk" (OuterVolumeSpecName: "kube-api-access-tghrk") pod "63cb1c90-d64b-4c40-ab92-7f430eebac3e" (UID: "63cb1c90-d64b-4c40-ab92-7f430eebac3e"). InnerVolumeSpecName "kube-api-access-tghrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:19:10.543419 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.543382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:10.626232 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.626189    2583 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63cb1c90-d64b-4c40-ab92-7f430eebac3e-cert\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:19:10.626232 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.626225    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tghrk\" (UniqueName: \"kubernetes.io/projected/63cb1c90-d64b-4c40-ab92-7f430eebac3e-kube-api-access-tghrk\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:19:10.813876 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:10.813840    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5fcd7899-kx7cf"]
Sep 17 23:19:10.817038 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:19:10.816987    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11faebd_ccf3_4927_909b_8cdbd4901abf.slice/crio-b2a6dad689ed923654ee64744273c70a1fb2004dc984a4666902863486dfcc17 WatchSource:0}: Error finding container b2a6dad689ed923654ee64744273c70a1fb2004dc984a4666902863486dfcc17: Status 404 returned error can't find the container with id b2a6dad689ed923654ee64744273c70a1fb2004dc984a4666902863486dfcc17
Sep 17 23:19:11.245402 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.245364    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf" event={"ID":"b11faebd-ccf3-4927-909b-8cdbd4901abf","Type":"ContainerStarted","Data":"b2a6dad689ed923654ee64744273c70a1fb2004dc984a4666902863486dfcc17"}
Sep 17 23:19:11.246646 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.246618    2583 generic.go:356] "Generic (PLEG): container finished" podID="63cb1c90-d64b-4c40-ab92-7f430eebac3e" containerID="3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188" exitCode=0
Sep 17 23:19:11.246734 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.246664    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq"
Sep 17 23:19:11.246734 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.246680    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq" event={"ID":"63cb1c90-d64b-4c40-ab92-7f430eebac3e","Type":"ContainerDied","Data":"3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188"}
Sep 17 23:19:11.246734 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.246725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5fcd7899-xs9tq" event={"ID":"63cb1c90-d64b-4c40-ab92-7f430eebac3e","Type":"ContainerDied","Data":"cfc582b721175cb6b6bb111ba4fea90512b0f433d26e3320f157618423ba6926"}
Sep 17 23:19:11.246838 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.246751    2583 scope.go:117] "RemoveContainer" containerID="3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188"
Sep 17 23:19:11.254981 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.254960    2583 scope.go:117] "RemoveContainer" containerID="3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188"
Sep 17 23:19:11.255240 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:19:11.255224    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188\": container with ID starting with 3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188 not found: ID does not exist" containerID="3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188"
Sep 17 23:19:11.255296 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.255248    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188"} err="failed to get container status \"3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188\": rpc error: code = NotFound desc = could not find container \"3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188\": container with ID starting with 3a8eede136bd9997e100e38d04d56134e196b103019e472ec3abeb5af2c6d188 not found: ID does not exist"
Sep 17 23:19:11.263563 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.263541    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5fcd7899-xs9tq"]
Sep 17 23:19:11.269661 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:11.269637    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5c5fcd7899-xs9tq"]
Sep 17 23:19:12.252930 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:12.252891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf" event={"ID":"b11faebd-ccf3-4927-909b-8cdbd4901abf","Type":"ContainerStarted","Data":"25850b6e9b055f56826b54b314f48958f4a85cd2387ec373dc85697c6be88c94"}
Sep 17 23:19:12.253419 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:12.252963    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:12.270571 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:12.270518    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf" podStartSLOduration=1.7586192139999999 podStartE2EDuration="2.27050025s" totalImagesPullingTime="511.881036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:12.270128767 +0000 UTC m=+515.204574519" watchObservedRunningTime="2026-09-17 23:19:12.27050025 +0000 UTC m=+515.204946038"
Sep 17 23:19:13.156866 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:13.156828    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63cb1c90-d64b-4c40-ab92-7f430eebac3e" path="/var/lib/kubelet/pods/63cb1c90-d64b-4c40-ab92-7f430eebac3e/volumes"
Sep 17 23:19:43.264085 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:43.264038    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5fcd7899-kx7cf"
Sep 17 23:19:44.557166 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.557123    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-444h2"]
Sep 17 23:19:44.557619 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.557448    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63cb1c90-d64b-4c40-ab92-7f430eebac3e" containerName="manager"
Sep 17 23:19:44.557619 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.557461    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="63cb1c90-d64b-4c40-ab92-7f430eebac3e" containerName="manager"
Sep 17 23:19:44.557619 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.557525    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="63cb1c90-d64b-4c40-ab92-7f430eebac3e" containerName="manager"
Sep 17 23:19:44.562817 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.562782    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:19:44.565395 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.565373    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-hsq27\""
Sep 17 23:19:44.565526 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.565402    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 23:19:44.571975 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.571950    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-444h2"]
Sep 17 23:19:44.704513 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.704474    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwkjx\" (UniqueName: \"kubernetes.io/projected/eb656762-5df2-487b-9852-3fbca0c02574-kube-api-access-pwkjx\") pod \"model-serving-api-7b954ffc95-444h2\" (UID: \"eb656762-5df2-487b-9852-3fbca0c02574\") " pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:19:44.704513 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.704515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb656762-5df2-487b-9852-3fbca0c02574-tls-certs\") pod \"model-serving-api-7b954ffc95-444h2\" (UID: \"eb656762-5df2-487b-9852-3fbca0c02574\") " pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:19:44.805668 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.805633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwkjx\" (UniqueName: \"kubernetes.io/projected/eb656762-5df2-487b-9852-3fbca0c02574-kube-api-access-pwkjx\") pod \"model-serving-api-7b954ffc95-444h2\" (UID: \"eb656762-5df2-487b-9852-3fbca0c02574\") " pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:19:44.805668 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.805671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb656762-5df2-487b-9852-3fbca0c02574-tls-certs\") pod \"model-serving-api-7b954ffc95-444h2\" (UID: \"eb656762-5df2-487b-9852-3fbca0c02574\") " pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:19:44.808299 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.808229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb656762-5df2-487b-9852-3fbca0c02574-tls-certs\") pod \"model-serving-api-7b954ffc95-444h2\" (UID: \"eb656762-5df2-487b-9852-3fbca0c02574\") " pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:19:44.814338 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.814311    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwkjx\" (UniqueName: \"kubernetes.io/projected/eb656762-5df2-487b-9852-3fbca0c02574-kube-api-access-pwkjx\") pod \"model-serving-api-7b954ffc95-444h2\" (UID: \"eb656762-5df2-487b-9852-3fbca0c02574\") " pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:19:44.874086 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:44.874044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:19:45.150013 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:45.149984    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-444h2"]
Sep 17 23:19:45.152636 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:19:45.152601    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb656762_5df2_487b_9852_3fbca0c02574.slice/crio-18bccd9f371484da7571f5eb1765f447a381e4d59de4b537613567b387b439a2 WatchSource:0}: Error finding container 18bccd9f371484da7571f5eb1765f447a381e4d59de4b537613567b387b439a2: Status 404 returned error can't find the container with id 18bccd9f371484da7571f5eb1765f447a381e4d59de4b537613567b387b439a2
Sep 17 23:19:45.393903 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:45.393863    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-444h2" event={"ID":"eb656762-5df2-487b-9852-3fbca0c02574","Type":"ContainerStarted","Data":"18bccd9f371484da7571f5eb1765f447a381e4d59de4b537613567b387b439a2"}
Sep 17 23:19:47.407287 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:47.407243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-444h2" event={"ID":"eb656762-5df2-487b-9852-3fbca0c02574","Type":"ContainerStarted","Data":"ed1480230dcb8761ef72d1e36187f24e1040c7c84966bddffe63413e5731d1ec"}
Sep 17 23:19:47.407759 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:47.407329    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:19:47.427783 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:47.427720    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-444h2" podStartSLOduration=1.293798347 podStartE2EDuration="3.427702327s" totalImagesPullingTime="2.13390398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:47.426076501 +0000 UTC m=+550.360522270" watchObservedRunningTime="2026-09-17 23:19:47.427702327 +0000 UTC m=+550.362148076"
Sep 17 23:19:58.416910 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:19:58.416878    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-444h2"
Sep 17 23:20:39.071409 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:20:39.071372    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:20:39.074671 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:20:39.074644    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:20:39.089383 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:20:39.089352    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:20:39.092656 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:20:39.092629    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:21:29.205721 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.205680    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"]
Sep 17 23:21:29.209385 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.209358    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.212052 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.212000    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b6ba0-predictor-serving-cert\""
Sep 17 23:21:29.212052 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.212007    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b6ba0-kube-rbac-proxy-sar-config\""
Sep 17 23:21:29.212275 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.212007    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:21:29.212275 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.212131    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9ls6r\""
Sep 17 23:21:29.212897 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.212877    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:21:29.220621 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.220390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa4e94-21d0-40bb-b318-115624a91754-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.220621 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.220459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa4e94-21d0-40bb-b318-115624a91754-proxy-tls\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.220621 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.220536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7t4\" (UniqueName: \"kubernetes.io/projected/76aa4e94-21d0-40bb-b318-115624a91754-kube-api-access-xh7t4\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.222335 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.222306    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"]
Sep 17 23:21:29.321154 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.321113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7t4\" (UniqueName: \"kubernetes.io/projected/76aa4e94-21d0-40bb-b318-115624a91754-kube-api-access-xh7t4\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.321387 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.321164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa4e94-21d0-40bb-b318-115624a91754-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.321387 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.321197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa4e94-21d0-40bb-b318-115624a91754-proxy-tls\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.321387 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:21:29.321300    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-serving-cert: secret "success-200-isvc-b6ba0-predictor-serving-cert" not found
Sep 17 23:21:29.321387 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:21:29.321364    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76aa4e94-21d0-40bb-b318-115624a91754-proxy-tls podName:76aa4e94-21d0-40bb-b318-115624a91754 nodeName:}" failed. No retries permitted until 2026-09-17 23:21:29.821349196 +0000 UTC m=+652.755794924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/76aa4e94-21d0-40bb-b318-115624a91754-proxy-tls") pod "success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" (UID: "76aa4e94-21d0-40bb-b318-115624a91754") : secret "success-200-isvc-b6ba0-predictor-serving-cert" not found
Sep 17 23:21:29.321922 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.321896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa4e94-21d0-40bb-b318-115624a91754-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.332878 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.332841    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7t4\" (UniqueName: \"kubernetes.io/projected/76aa4e94-21d0-40bb-b318-115624a91754-kube-api-access-xh7t4\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.826375 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.826325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa4e94-21d0-40bb-b318-115624a91754-proxy-tls\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:29.829127 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:29.829099    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa4e94-21d0-40bb-b318-115624a91754-proxy-tls\") pod \"success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") " pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:30.121587 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:30.121486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:30.399718 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:30.399688    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"]
Sep 17 23:21:30.402406 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:21:30.402375    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76aa4e94_21d0_40bb_b318_115624a91754.slice/crio-45f963c585f3d8ca60bf7c5387a6483a914d64bc7fd3e089b63672a4070aeac4 WatchSource:0}: Error finding container 45f963c585f3d8ca60bf7c5387a6483a914d64bc7fd3e089b63672a4070aeac4: Status 404 returned error can't find the container with id 45f963c585f3d8ca60bf7c5387a6483a914d64bc7fd3e089b63672a4070aeac4
Sep 17 23:21:30.872300 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:30.872261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" event={"ID":"76aa4e94-21d0-40bb-b318-115624a91754","Type":"ContainerStarted","Data":"45f963c585f3d8ca60bf7c5387a6483a914d64bc7fd3e089b63672a4070aeac4"}
Sep 17 23:21:42.403461 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:42.403440    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:21:42.942335 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:42.942287    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" event={"ID":"76aa4e94-21d0-40bb-b318-115624a91754","Type":"ContainerStarted","Data":"cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751"}
Sep 17 23:21:44.954068 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:44.954000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" event={"ID":"76aa4e94-21d0-40bb-b318-115624a91754","Type":"ContainerStarted","Data":"1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad"}
Sep 17 23:21:44.954538 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:44.954144    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:44.954538 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:44.954182    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:44.955868 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:44.955837    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:21:44.971924 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:44.971863    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podStartSLOduration=1.696454594 podStartE2EDuration="15.971846701s" totalImagesPullingTime="14.275392107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:44.971398266 +0000 UTC m=+667.905844018" watchObservedRunningTime="2026-09-17 23:21:44.971846701 +0000 UTC m=+667.906292450"
Sep 17 23:21:45.958767 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:45.958718    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:21:50.963159 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:50.963122    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:21:50.963769 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:21:50.963742    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:22:00.964614 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:00.964571    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:22:10.964283 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:10.964242    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:22:20.964712 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:20.964669    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 23:22:30.965415 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:30.965378    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:22:48.942440 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:48.942396    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"]
Sep 17 23:22:48.946363 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:48.946331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:48.949004 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:48.948972    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b6ba0-serving-cert\""
Sep 17 23:22:48.949202 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:48.948972    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-b6ba0-kube-rbac-proxy-sar-config\""
Sep 17 23:22:48.955728 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:48.955699    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"]
Sep 17 23:22:49.003093 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:49.003040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls\") pod \"switch-graph-b6ba0-69bcc565d7-ct95z\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") " pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:49.003350 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:49.003158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9844c5-fd16-4bee-8a28-94212e619177-openshift-service-ca-bundle\") pod \"switch-graph-b6ba0-69bcc565d7-ct95z\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") " pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:49.104604 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:49.104535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9844c5-fd16-4bee-8a28-94212e619177-openshift-service-ca-bundle\") pod \"switch-graph-b6ba0-69bcc565d7-ct95z\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") " pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:49.104829 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:49.104707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls\") pod \"switch-graph-b6ba0-69bcc565d7-ct95z\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") " pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:49.104902 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:22:49.104838    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-b6ba0-serving-cert: secret "switch-graph-b6ba0-serving-cert" not found
Sep 17 23:22:49.104954 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:22:49.104924    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls podName:bf9844c5-fd16-4bee-8a28-94212e619177 nodeName:}" failed. No retries permitted until 2026-09-17 23:22:49.604900091 +0000 UTC m=+732.539345838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls") pod "switch-graph-b6ba0-69bcc565d7-ct95z" (UID: "bf9844c5-fd16-4bee-8a28-94212e619177") : secret "switch-graph-b6ba0-serving-cert" not found
Sep 17 23:22:49.105370 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:49.105345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9844c5-fd16-4bee-8a28-94212e619177-openshift-service-ca-bundle\") pod \"switch-graph-b6ba0-69bcc565d7-ct95z\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") " pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:49.608529 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:49.608488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls\") pod \"switch-graph-b6ba0-69bcc565d7-ct95z\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") " pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:49.608752 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:22:49.608663    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-b6ba0-serving-cert: secret "switch-graph-b6ba0-serving-cert" not found
Sep 17 23:22:49.608837 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:22:49.608756    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls podName:bf9844c5-fd16-4bee-8a28-94212e619177 nodeName:}" failed. No retries permitted until 2026-09-17 23:22:50.608733891 +0000 UTC m=+733.543179622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls") pod "switch-graph-b6ba0-69bcc565d7-ct95z" (UID: "bf9844c5-fd16-4bee-8a28-94212e619177") : secret "switch-graph-b6ba0-serving-cert" not found
Sep 17 23:22:50.618441 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:50.618398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls\") pod \"switch-graph-b6ba0-69bcc565d7-ct95z\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") " pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:50.620953 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:50.620924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls\") pod \"switch-graph-b6ba0-69bcc565d7-ct95z\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") " pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:50.758900 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:50.758854    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:51.031488 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:51.031461    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"]
Sep 17 23:22:51.034351 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:22:51.034309    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9844c5_fd16_4bee_8a28_94212e619177.slice/crio-18ef4f952427faff48c58a34a573d21263d30c6c2e8747460f2c0c0db8e034e9 WatchSource:0}: Error finding container 18ef4f952427faff48c58a34a573d21263d30c6c2e8747460f2c0c0db8e034e9: Status 404 returned error can't find the container with id 18ef4f952427faff48c58a34a573d21263d30c6c2e8747460f2c0c0db8e034e9
Sep 17 23:22:51.252579 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:51.252543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" event={"ID":"bf9844c5-fd16-4bee-8a28-94212e619177","Type":"ContainerStarted","Data":"18ef4f952427faff48c58a34a573d21263d30c6c2e8747460f2c0c0db8e034e9"}
Sep 17 23:22:54.268387 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:54.268342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" event={"ID":"bf9844c5-fd16-4bee-8a28-94212e619177","Type":"ContainerStarted","Data":"8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016"}
Sep 17 23:22:54.268795 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:54.268426    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:22:54.287300 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:22:54.287249    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" podStartSLOduration=3.992690962 podStartE2EDuration="6.287232078s" totalImagesPullingTime="2.294541116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:54.286262658 +0000 UTC m=+737.220708421" watchObservedRunningTime="2026-09-17 23:22:54.287232078 +0000 UTC m=+737.221677827"
Sep 17 23:23:00.278843 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:00.278815    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:23:03.082265 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.082232    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"]
Sep 17 23:23:03.082798 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.082450    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0" containerID="cri-o://8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016" gracePeriod=30
Sep 17 23:23:03.263278 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.263242    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"]
Sep 17 23:23:03.263594 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.263561    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container" containerID="cri-o://cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751" gracePeriod=30
Sep 17 23:23:03.263695 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.263653    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kube-rbac-proxy" containerID="cri-o://1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad" gracePeriod=30
Sep 17 23:23:03.449535 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.449498    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"]
Sep 17 23:23:03.452905 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.452889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.455739 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.455714    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-11096-kube-rbac-proxy-sar-config\""
Sep 17 23:23:03.455739 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.455714    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-11096-predictor-serving-cert\""
Sep 17 23:23:03.467486 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.467460    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"]
Sep 17 23:23:03.521761 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.521727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-11096-predictor-587bc899cb-dtvwk\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") " pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.521929 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.521796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsv4t\" (UniqueName: \"kubernetes.io/projected/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-api-access-hsv4t\") pod \"success-200-isvc-11096-predictor-587bc899cb-dtvwk\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") " pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.521929 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.521848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbe12c0-40dc-4b43-884f-1680d643c069-proxy-tls\") pod \"success-200-isvc-11096-predictor-587bc899cb-dtvwk\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") " pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.623245 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.623206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-11096-predictor-587bc899cb-dtvwk\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") " pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.623492 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.623259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsv4t\" (UniqueName: \"kubernetes.io/projected/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-api-access-hsv4t\") pod \"success-200-isvc-11096-predictor-587bc899cb-dtvwk\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") " pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.623492 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.623298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbe12c0-40dc-4b43-884f-1680d643c069-proxy-tls\") pod \"success-200-isvc-11096-predictor-587bc899cb-dtvwk\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") " pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.623901 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.623878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-11096-predictor-587bc899cb-dtvwk\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") " pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.625921 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.625893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbe12c0-40dc-4b43-884f-1680d643c069-proxy-tls\") pod \"success-200-isvc-11096-predictor-587bc899cb-dtvwk\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") " pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.643447 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.643385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsv4t\" (UniqueName: \"kubernetes.io/projected/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-api-access-hsv4t\") pod \"success-200-isvc-11096-predictor-587bc899cb-dtvwk\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") " pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:03.762863 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:03.762824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:04.036323 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.036295    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"]
Sep 17 23:23:04.038459 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:23:04.038432    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbe12c0_40dc_4b43_884f_1680d643c069.slice/crio-fac048eccae8b5af47ce382f866ae00623743c629de9109b8c655876f1a0ed60 WatchSource:0}: Error finding container fac048eccae8b5af47ce382f866ae00623743c629de9109b8c655876f1a0ed60: Status 404 returned error can't find the container with id fac048eccae8b5af47ce382f866ae00623743c629de9109b8c655876f1a0ed60
Sep 17 23:23:04.317069 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.316932    2583 generic.go:356] "Generic (PLEG): container finished" podID="76aa4e94-21d0-40bb-b318-115624a91754" containerID="1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad" exitCode=2
Sep 17 23:23:04.317069 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.316979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" event={"ID":"76aa4e94-21d0-40bb-b318-115624a91754","Type":"ContainerDied","Data":"1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad"}
Sep 17 23:23:04.319428 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.319401    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" event={"ID":"ccbe12c0-40dc-4b43-884f-1680d643c069","Type":"ContainerStarted","Data":"8a839b146d51eaaa96edb7a82929894fe6efc974c158a71a7c4f58bf8ecca3b1"}
Sep 17 23:23:04.319554 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.319459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" event={"ID":"ccbe12c0-40dc-4b43-884f-1680d643c069","Type":"ContainerStarted","Data":"c24b051b7c95b11a1824bfbedccd2afd4fcf0c70d4e30a7eae7dc01bc6286f99"}
Sep 17 23:23:04.319554 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.319487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" event={"ID":"ccbe12c0-40dc-4b43-884f-1680d643c069","Type":"ContainerStarted","Data":"fac048eccae8b5af47ce382f866ae00623743c629de9109b8c655876f1a0ed60"}
Sep 17 23:23:04.319972 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.319779    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:04.319972 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.319842    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:04.321851 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.321822    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:23:04.342341 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:04.342298    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podStartSLOduration=1.3422809199999999 podStartE2EDuration="1.34228092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:04.340593533 +0000 UTC m=+747.275039321" watchObservedRunningTime="2026-09-17 23:23:04.34228092 +0000 UTC m=+747.276726670"
Sep 17 23:23:05.276725 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:05.276678    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:05.325092 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:05.325049    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:23:05.959418 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:05.959375    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 23:23:06.512252 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.512227    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:23:06.644904 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.644814    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa4e94-21d0-40bb-b318-115624a91754-kube-rbac-proxy-sar-config\") pod \"76aa4e94-21d0-40bb-b318-115624a91754\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") "
Sep 17 23:23:06.644904 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.644873    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh7t4\" (UniqueName: \"kubernetes.io/projected/76aa4e94-21d0-40bb-b318-115624a91754-kube-api-access-xh7t4\") pod \"76aa4e94-21d0-40bb-b318-115624a91754\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") "
Sep 17 23:23:06.645163 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.644913    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa4e94-21d0-40bb-b318-115624a91754-proxy-tls\") pod \"76aa4e94-21d0-40bb-b318-115624a91754\" (UID: \"76aa4e94-21d0-40bb-b318-115624a91754\") "
Sep 17 23:23:06.645231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.645172    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76aa4e94-21d0-40bb-b318-115624a91754-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76aa4e94-21d0-40bb-b318-115624a91754" (UID: "76aa4e94-21d0-40bb-b318-115624a91754"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:06.647109 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.647080    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76aa4e94-21d0-40bb-b318-115624a91754-kube-api-access-xh7t4" (OuterVolumeSpecName: "kube-api-access-xh7t4") pod "76aa4e94-21d0-40bb-b318-115624a91754" (UID: "76aa4e94-21d0-40bb-b318-115624a91754"). InnerVolumeSpecName "kube-api-access-xh7t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:23:06.647225 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.647131    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76aa4e94-21d0-40bb-b318-115624a91754-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76aa4e94-21d0-40bb-b318-115624a91754" (UID: "76aa4e94-21d0-40bb-b318-115624a91754"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:06.746236 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.746200    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76aa4e94-21d0-40bb-b318-115624a91754-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:23:06.746236 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.746229    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7t4\" (UniqueName: \"kubernetes.io/projected/76aa4e94-21d0-40bb-b318-115624a91754-kube-api-access-xh7t4\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:23:06.746236 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:06.746239    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76aa4e94-21d0-40bb-b318-115624a91754-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:23:07.334903 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.334867    2583 generic.go:356] "Generic (PLEG): container finished" podID="76aa4e94-21d0-40bb-b318-115624a91754" containerID="cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751" exitCode=0
Sep 17 23:23:07.335108 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.334921    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"
Sep 17 23:23:07.335108 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.334942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" event={"ID":"76aa4e94-21d0-40bb-b318-115624a91754","Type":"ContainerDied","Data":"cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751"}
Sep 17 23:23:07.335108 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.334998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t" event={"ID":"76aa4e94-21d0-40bb-b318-115624a91754","Type":"ContainerDied","Data":"45f963c585f3d8ca60bf7c5387a6483a914d64bc7fd3e089b63672a4070aeac4"}
Sep 17 23:23:07.335108 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.335046    2583 scope.go:117] "RemoveContainer" containerID="1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad"
Sep 17 23:23:07.343546 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.343528    2583 scope.go:117] "RemoveContainer" containerID="cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751"
Sep 17 23:23:07.350785 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.350767    2583 scope.go:117] "RemoveContainer" containerID="1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad"
Sep 17 23:23:07.351097 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:23:07.351075    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad\": container with ID starting with 1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad not found: ID does not exist" containerID="1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad"
Sep 17 23:23:07.351156 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.351106    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad"} err="failed to get container status \"1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad\": rpc error: code = NotFound desc = could not find container \"1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad\": container with ID starting with 1a2055335fc914f3a25c764dba1fd61903f924db001cdfe4e854b5184f0b26ad not found: ID does not exist"
Sep 17 23:23:07.351156 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.351122    2583 scope.go:117] "RemoveContainer" containerID="cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751"
Sep 17 23:23:07.351431 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:23:07.351389    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751\": container with ID starting with cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751 not found: ID does not exist" containerID="cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751"
Sep 17 23:23:07.351431 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.351428    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751"} err="failed to get container status \"cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751\": rpc error: code = NotFound desc = could not find container \"cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751\": container with ID starting with cc14104fa892713bf7c0cfd37e5aa2c4c95815850fc51d4061eacc52be750751 not found: ID does not exist"
Sep 17 23:23:07.354069 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.354049    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"]
Sep 17 23:23:07.358902 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:07.358881    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b6ba0-predictor-76bc585db8-ldc2t"]
Sep 17 23:23:09.156890 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:09.156859    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76aa4e94-21d0-40bb-b318-115624a91754" path="/var/lib/kubelet/pods/76aa4e94-21d0-40bb-b318-115624a91754/volumes"
Sep 17 23:23:10.277067 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:10.277004    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:10.329456 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:10.329430    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:10.329923 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:10.329899    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:23:15.276450 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:15.276407    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:15.276866 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:15.276516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:23:20.277113 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:20.277076    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:20.330255 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:20.330221    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:23:25.277516 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:25.277463    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:30.277147 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:30.277111    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:30.330150 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:30.330115    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:23:33.281944 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.281919    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:23:33.447360 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.447274    2583 generic.go:356] "Generic (PLEG): container finished" podID="bf9844c5-fd16-4bee-8a28-94212e619177" containerID="8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016" exitCode=0
Sep 17 23:23:33.447360 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.447324    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"
Sep 17 23:23:33.447548 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.447361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" event={"ID":"bf9844c5-fd16-4bee-8a28-94212e619177","Type":"ContainerDied","Data":"8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016"}
Sep 17 23:23:33.447548 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.447415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z" event={"ID":"bf9844c5-fd16-4bee-8a28-94212e619177","Type":"ContainerDied","Data":"18ef4f952427faff48c58a34a573d21263d30c6c2e8747460f2c0c0db8e034e9"}
Sep 17 23:23:33.447548 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.447439    2583 scope.go:117] "RemoveContainer" containerID="8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016"
Sep 17 23:23:33.455242 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.455225    2583 scope.go:117] "RemoveContainer" containerID="8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016"
Sep 17 23:23:33.455489 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:23:33.455470    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016\": container with ID starting with 8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016 not found: ID does not exist" containerID="8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016"
Sep 17 23:23:33.455534 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.455498    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016"} err="failed to get container status \"8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016\": rpc error: code = NotFound desc = could not find container \"8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016\": container with ID starting with 8ba27c345bcca6c8842999b0bd499a1891fb749e5203f6e4eaa4398c8e7df016 not found: ID does not exist"
Sep 17 23:23:33.464779 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.464757    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls\") pod \"bf9844c5-fd16-4bee-8a28-94212e619177\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") "
Sep 17 23:23:33.464846 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.464800    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9844c5-fd16-4bee-8a28-94212e619177-openshift-service-ca-bundle\") pod \"bf9844c5-fd16-4bee-8a28-94212e619177\" (UID: \"bf9844c5-fd16-4bee-8a28-94212e619177\") "
Sep 17 23:23:33.465169 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.465143    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9844c5-fd16-4bee-8a28-94212e619177-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bf9844c5-fd16-4bee-8a28-94212e619177" (UID: "bf9844c5-fd16-4bee-8a28-94212e619177"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:23:33.466630 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.466612    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf9844c5-fd16-4bee-8a28-94212e619177" (UID: "bf9844c5-fd16-4bee-8a28-94212e619177"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:23:33.565957 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.565930    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf9844c5-fd16-4bee-8a28-94212e619177-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:23:33.565957 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.565956    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf9844c5-fd16-4bee-8a28-94212e619177-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:23:33.768551 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.768523    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"]
Sep 17 23:23:33.772452 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:33.772430    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-b6ba0-69bcc565d7-ct95z"]
Sep 17 23:23:35.157778 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:35.157745    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" path="/var/lib/kubelet/pods/bf9844c5-fd16-4bee-8a28-94212e619177/volumes"
Sep 17 23:23:38.981795 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.981758    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"]
Sep 17 23:23:38.982302 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.982169    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kube-rbac-proxy"
Sep 17 23:23:38.982302 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.982184    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kube-rbac-proxy"
Sep 17 23:23:38.982302 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.982191    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0"
Sep 17 23:23:38.982302 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.982196    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0"
Sep 17 23:23:38.982302 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.982205    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container"
Sep 17 23:23:38.982302 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.982212    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container"
Sep 17 23:23:38.982302 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.982277    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kube-rbac-proxy"
Sep 17 23:23:38.982302 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.982298    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="76aa4e94-21d0-40bb-b318-115624a91754" containerName="kserve-container"
Sep 17 23:23:38.982302 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.982303    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf9844c5-fd16-4bee-8a28-94212e619177" containerName="switch-graph-b6ba0"
Sep 17 23:23:38.986343 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.986326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:38.989002 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.988980    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 17 23:23:38.989275 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.989254    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Sep 17 23:23:38.999093 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:38.999068    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"]
Sep 17 23:23:39.108869 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:39.108835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-proxy-tls\") pod \"model-chainer-5c78d4956c-gjrh7\" (UID: \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\") " pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:39.109039 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:39.108902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-openshift-service-ca-bundle\") pod \"model-chainer-5c78d4956c-gjrh7\" (UID: \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\") " pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:39.209879 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:39.209851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-proxy-tls\") pod \"model-chainer-5c78d4956c-gjrh7\" (UID: \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\") " pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:39.209989 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:39.209909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-openshift-service-ca-bundle\") pod \"model-chainer-5c78d4956c-gjrh7\" (UID: \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\") " pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:39.210441 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:39.210423    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-openshift-service-ca-bundle\") pod \"model-chainer-5c78d4956c-gjrh7\" (UID: \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\") " pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:39.212631 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:39.212616    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Sep 17 23:23:39.223196 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:39.223164    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-proxy-tls\") pod \"model-chainer-5c78d4956c-gjrh7\" (UID: \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\") " pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:39.296343 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:39.296322    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:39.566803 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:39.566720    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"]
Sep 17 23:23:39.568920 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:23:39.568888    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3365ee42_d2b4_4fd9_a435_d03b7ebdd036.slice/crio-7e45a4049789c775b8dd7cf4b2ddfb0a700069aa66af6b84ec71cf23966ab28a WatchSource:0}: Error finding container 7e45a4049789c775b8dd7cf4b2ddfb0a700069aa66af6b84ec71cf23966ab28a: Status 404 returned error can't find the container with id 7e45a4049789c775b8dd7cf4b2ddfb0a700069aa66af6b84ec71cf23966ab28a
Sep 17 23:23:40.330479 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:40.330443    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:23:40.478472 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:40.478440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" event={"ID":"3365ee42-d2b4-4fd9-a435-d03b7ebdd036","Type":"ContainerStarted","Data":"166f3717d8acdf1e7d6978632a77cc919f467e32b6402cfc903cb4292c8165ca"}
Sep 17 23:23:40.478636 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:40.478493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" event={"ID":"3365ee42-d2b4-4fd9-a435-d03b7ebdd036","Type":"ContainerStarted","Data":"7e45a4049789c775b8dd7cf4b2ddfb0a700069aa66af6b84ec71cf23966ab28a"}
Sep 17 23:23:40.478636 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:40.478564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:40.495785 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:40.495741    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" podStartSLOduration=2.495726555 podStartE2EDuration="2.495726555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:40.494791428 +0000 UTC m=+783.429237177" watchObservedRunningTime="2026-09-17 23:23:40.495726555 +0000 UTC m=+783.430172305"
Sep 17 23:23:46.487786 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:46.487757    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:23:49.618855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:49.618816    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"]
Sep 17 23:23:49.619392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:49.619098    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer" containerID="cri-o://166f3717d8acdf1e7d6978632a77cc919f467e32b6402cfc903cb4292c8165ca" gracePeriod=30
Sep 17 23:23:50.069694 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.069655    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"]
Sep 17 23:23:50.072983 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.072964    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.075866 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.075848    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2026c-predictor-serving-cert\""
Sep 17 23:23:50.076159 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.076138    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2026c-kube-rbac-proxy-sar-config\""
Sep 17 23:23:50.085643 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.085621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20639def-b323-4067-ade5-b36de99fa233-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.085732 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.085652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpsps\" (UniqueName: \"kubernetes.io/projected/20639def-b323-4067-ade5-b36de99fa233-kube-api-access-cpsps\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.085732 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.085684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.090670 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.090651    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"]
Sep 17 23:23:50.186770 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.186734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsps\" (UniqueName: \"kubernetes.io/projected/20639def-b323-4067-ade5-b36de99fa233-kube-api-access-cpsps\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.186944 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.186799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.186944 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.186876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20639def-b323-4067-ade5-b36de99fa233-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.186944 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:23:50.186892    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-2026c-predictor-serving-cert: secret "success-200-isvc-2026c-predictor-serving-cert" not found
Sep 17 23:23:50.187150 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:23:50.186968    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls podName:20639def-b323-4067-ade5-b36de99fa233 nodeName:}" failed. No retries permitted until 2026-09-17 23:23:50.686947742 +0000 UTC m=+793.621393483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls") pod "success-200-isvc-2026c-predictor-866c975b8d-dr7dv" (UID: "20639def-b323-4067-ade5-b36de99fa233") : secret "success-200-isvc-2026c-predictor-serving-cert" not found
Sep 17 23:23:50.187630 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.187607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20639def-b323-4067-ade5-b36de99fa233-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.209410 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.209384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpsps\" (UniqueName: \"kubernetes.io/projected/20639def-b323-4067-ade5-b36de99fa233-kube-api-access-cpsps\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.331203 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.331146    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:23:50.690455 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:50.690379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:50.690807 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:23:50.690494    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-2026c-predictor-serving-cert: secret "success-200-isvc-2026c-predictor-serving-cert" not found
Sep 17 23:23:50.690807 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:23:50.690569    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls podName:20639def-b323-4067-ade5-b36de99fa233 nodeName:}" failed. No retries permitted until 2026-09-17 23:23:51.690554444 +0000 UTC m=+794.625000172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls") pod "success-200-isvc-2026c-predictor-866c975b8d-dr7dv" (UID: "20639def-b323-4067-ade5-b36de99fa233") : secret "success-200-isvc-2026c-predictor-serving-cert" not found
Sep 17 23:23:51.486208 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:51.486128    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:51.697031 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:51.696996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:51.699367 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:51.699343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls\") pod \"success-200-isvc-2026c-predictor-866c975b8d-dr7dv\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") " pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:51.883167 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:51.883137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:52.168673 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:52.168644    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"]
Sep 17 23:23:52.531296 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:52.531256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" event={"ID":"20639def-b323-4067-ade5-b36de99fa233","Type":"ContainerStarted","Data":"ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b"}
Sep 17 23:23:52.531449 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:52.531321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" event={"ID":"20639def-b323-4067-ade5-b36de99fa233","Type":"ContainerStarted","Data":"394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968"}
Sep 17 23:23:52.531449 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:52.531353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" event={"ID":"20639def-b323-4067-ade5-b36de99fa233","Type":"ContainerStarted","Data":"723dab41925e0066b48407047dabcc021bc7c979f098140c87bfc8ab2835cf70"}
Sep 17 23:23:52.531449 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:52.531422    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:52.553642 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:52.553599    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" podStartSLOduration=2.553587256 podStartE2EDuration="2.553587256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:52.553295743 +0000 UTC m=+795.487741494" watchObservedRunningTime="2026-09-17 23:23:52.553587256 +0000 UTC m=+795.488033006"
Sep 17 23:23:53.535457 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:53.535424    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:53.536675 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:53.536649    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:23:54.539849 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:54.539810    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:23:56.487034 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:56.486994    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:59.544814 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:59.544784    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:23:59.545275 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:23:59.545207    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:24:01.486904 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:01.486865    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:01.487293 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:01.486970    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:24:03.388268 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.388236    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"]
Sep 17 23:24:03.391503 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.391485    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:03.398361 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.398330    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-11096-serving-cert\""
Sep 17 23:24:03.398469 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.398375    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-11096-kube-rbac-proxy-sar-config\""
Sep 17 23:24:03.470549 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.470522    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"]
Sep 17 23:24:03.478570 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.478545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-proxy-tls\") pod \"switch-graph-11096-d974cf6bd-77wlw\" (UID: \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\") " pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:03.478690 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.478591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-openshift-service-ca-bundle\") pod \"switch-graph-11096-d974cf6bd-77wlw\" (UID: \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\") " pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:03.578966 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.578938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-proxy-tls\") pod \"switch-graph-11096-d974cf6bd-77wlw\" (UID: \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\") " pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:03.579119 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.578978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-openshift-service-ca-bundle\") pod \"switch-graph-11096-d974cf6bd-77wlw\" (UID: \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\") " pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:03.579186 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:24:03.579121    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-11096-serving-cert: secret "switch-graph-11096-serving-cert" not found
Sep 17 23:24:03.579225 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:24:03.579196    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-proxy-tls podName:9ae51a90-c525-49e0-a1f8-d2c95f7194b9 nodeName:}" failed. No retries permitted until 2026-09-17 23:24:04.079174092 +0000 UTC m=+807.013619832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-proxy-tls") pod "switch-graph-11096-d974cf6bd-77wlw" (UID: "9ae51a90-c525-49e0-a1f8-d2c95f7194b9") : secret "switch-graph-11096-serving-cert" not found
Sep 17 23:24:03.579558 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:03.579541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-openshift-service-ca-bundle\") pod \"switch-graph-11096-d974cf6bd-77wlw\" (UID: \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\") " pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:04.083600 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:04.083570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-proxy-tls\") pod \"switch-graph-11096-d974cf6bd-77wlw\" (UID: \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\") " pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:04.085934 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:04.085906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-proxy-tls\") pod \"switch-graph-11096-d974cf6bd-77wlw\" (UID: \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\") " pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:04.301496 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:04.301468    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:04.574084 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:04.574051    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"]
Sep 17 23:24:04.576330 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:24:04.576304    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae51a90_c525_49e0_a1f8_d2c95f7194b9.slice/crio-601e9c8ae297d13c5268c74b648eeba7a94900f196291bdba42b0b6fecf55565 WatchSource:0}: Error finding container 601e9c8ae297d13c5268c74b648eeba7a94900f196291bdba42b0b6fecf55565: Status 404 returned error can't find the container with id 601e9c8ae297d13c5268c74b648eeba7a94900f196291bdba42b0b6fecf55565
Sep 17 23:24:04.587173 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:04.587140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" event={"ID":"9ae51a90-c525-49e0-a1f8-d2c95f7194b9","Type":"ContainerStarted","Data":"601e9c8ae297d13c5268c74b648eeba7a94900f196291bdba42b0b6fecf55565"}
Sep 17 23:24:05.593390 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:05.593357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" event={"ID":"9ae51a90-c525-49e0-a1f8-d2c95f7194b9","Type":"ContainerStarted","Data":"a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f"}
Sep 17 23:24:05.593768 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:05.593552    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:05.610708 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:05.610664    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" podStartSLOduration=2.6106505970000002 podStartE2EDuration="2.610650597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:05.609711823 +0000 UTC m=+808.544157572" watchObservedRunningTime="2026-09-17 23:24:05.610650597 +0000 UTC m=+808.545096346"
Sep 17 23:24:06.487066 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:06.487007    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:09.545266 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:09.545222    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:24:11.487046 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:11.486988    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:11.603210 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:11.603180    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:24:16.486244 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:16.486208    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:19.545948 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.545913    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:24:19.654455 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.654416    2583 generic.go:356] "Generic (PLEG): container finished" podID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerID="166f3717d8acdf1e7d6978632a77cc919f467e32b6402cfc903cb4292c8165ca" exitCode=0
Sep 17 23:24:19.654794 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.654484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" event={"ID":"3365ee42-d2b4-4fd9-a435-d03b7ebdd036","Type":"ContainerDied","Data":"166f3717d8acdf1e7d6978632a77cc919f467e32b6402cfc903cb4292c8165ca"}
Sep 17 23:24:19.763054 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.763009    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:24:19.800515 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.800454    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-proxy-tls\") pod \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\" (UID: \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\") "
Sep 17 23:24:19.800515 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.800512    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-openshift-service-ca-bundle\") pod \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\" (UID: \"3365ee42-d2b4-4fd9-a435-d03b7ebdd036\") "
Sep 17 23:24:19.800884 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.800862    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3365ee42-d2b4-4fd9-a435-d03b7ebdd036" (UID: "3365ee42-d2b4-4fd9-a435-d03b7ebdd036"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:19.802519 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.802494    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3365ee42-d2b4-4fd9-a435-d03b7ebdd036" (UID: "3365ee42-d2b4-4fd9-a435-d03b7ebdd036"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:24:19.901213 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.901184    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:24:19.901213 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:19.901209    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3365ee42-d2b4-4fd9-a435-d03b7ebdd036-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:24:20.661068 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:20.661015    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"
Sep 17 23:24:20.661068 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:20.661044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7" event={"ID":"3365ee42-d2b4-4fd9-a435-d03b7ebdd036","Type":"ContainerDied","Data":"7e45a4049789c775b8dd7cf4b2ddfb0a700069aa66af6b84ec71cf23966ab28a"}
Sep 17 23:24:20.661631 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:20.661099    2583 scope.go:117] "RemoveContainer" containerID="166f3717d8acdf1e7d6978632a77cc919f467e32b6402cfc903cb4292c8165ca"
Sep 17 23:24:20.692275 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:20.692248    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"]
Sep 17 23:24:20.694812 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:20.694792    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5c78d4956c-gjrh7"]
Sep 17 23:24:21.156777 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:21.156746    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" path="/var/lib/kubelet/pods/3365ee42-d2b4-4fd9-a435-d03b7ebdd036/volumes"
Sep 17 23:24:29.545272 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:29.545234    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 23:24:39.547043 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:39.546992    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:24:49.840337 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.840304    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"]
Sep 17 23:24:49.840692 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.840665    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer"
Sep 17 23:24:49.840692 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.840679    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer"
Sep 17 23:24:49.840771 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.840759    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3365ee42-d2b4-4fd9-a435-d03b7ebdd036" containerName="model-chainer"
Sep 17 23:24:49.844999 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.844980    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:24:49.847538 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.847519    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2026c-serving-cert\""
Sep 17 23:24:49.848038 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.848012    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2026c-kube-rbac-proxy-sar-config\""
Sep 17 23:24:49.855551 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.855530    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"]
Sep 17 23:24:49.919801 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.919783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-proxy-tls\") pod \"sequence-graph-2026c-6498f87569-5x5bj\" (UID: \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\") " pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:24:49.919895 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:49.919829    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-openshift-service-ca-bundle\") pod \"sequence-graph-2026c-6498f87569-5x5bj\" (UID: \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\") " pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:24:50.020335 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.020303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-proxy-tls\") pod \"sequence-graph-2026c-6498f87569-5x5bj\" (UID: \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\") " pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:24:50.020473 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.020343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-openshift-service-ca-bundle\") pod \"sequence-graph-2026c-6498f87569-5x5bj\" (UID: \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\") " pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:24:50.021102 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.021082    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-openshift-service-ca-bundle\") pod \"sequence-graph-2026c-6498f87569-5x5bj\" (UID: \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\") " pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:24:50.022669 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.022652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-proxy-tls\") pod \"sequence-graph-2026c-6498f87569-5x5bj\" (UID: \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\") " pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:24:50.156061 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.156004    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:24:50.424853 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.424829    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"]
Sep 17 23:24:50.427089 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:24:50.427058    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f608d0b_e29e_4f6f_b8e0_edb51c43fa7e.slice/crio-ae5b6e767b5ae5660d153ded346d1ce95491fb939faad4e3acee2e47914bb16d WatchSource:0}: Error finding container ae5b6e767b5ae5660d153ded346d1ce95491fb939faad4e3acee2e47914bb16d: Status 404 returned error can't find the container with id ae5b6e767b5ae5660d153ded346d1ce95491fb939faad4e3acee2e47914bb16d
Sep 17 23:24:50.793583 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.793552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" event={"ID":"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e","Type":"ContainerStarted","Data":"5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034"}
Sep 17 23:24:50.793720 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.793603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" event={"ID":"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e","Type":"ContainerStarted","Data":"ae5b6e767b5ae5660d153ded346d1ce95491fb939faad4e3acee2e47914bb16d"}
Sep 17 23:24:50.793720 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.793640    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:24:50.810790 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:50.810737    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" podStartSLOduration=1.810721292 podStartE2EDuration="1.810721292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:50.810159149 +0000 UTC m=+853.744604900" watchObservedRunningTime="2026-09-17 23:24:50.810721292 +0000 UTC m=+853.745167043"
Sep 17 23:24:56.802940 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:24:56.802913    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:25:39.099321 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:25:39.099297    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:25:39.102619 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:25:39.102598    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:25:39.115272 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:25:39.115249    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:25:39.118267 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:25:39.118248    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:30:39.134797 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:30:39.134772    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:30:39.137985 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:30:39.137962    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:30:39.141324 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:30:39.141305    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:30:39.144316 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:30:39.144299    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:32:18.007391 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.007348    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"]
Sep 17 23:32:18.009977 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.007647    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096" containerID="cri-o://a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f" gracePeriod=30
Sep 17 23:32:18.140207 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.140162    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"]
Sep 17 23:32:18.140467 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.140438    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container" containerID="cri-o://c24b051b7c95b11a1824bfbedccd2afd4fcf0c70d4e30a7eae7dc01bc6286f99" gracePeriod=30
Sep 17 23:32:18.140580 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.140493    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kube-rbac-proxy" containerID="cri-o://8a839b146d51eaaa96edb7a82929894fe6efc974c158a71a7c4f58bf8ecca3b1" gracePeriod=30
Sep 17 23:32:18.291850 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.291819    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"]
Sep 17 23:32:18.295231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.295215    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.297549 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.297527    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3e634-predictor-serving-cert\""
Sep 17 23:32:18.297940 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.297921    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3e634-kube-rbac-proxy-sar-config\""
Sep 17 23:32:18.306471 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.306448    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"]
Sep 17 23:32:18.359118 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.359083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c20f3-8c32-4da8-81b2-9052585f616f-proxy-tls\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.359224 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.359152    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c20f3-8c32-4da8-81b2-9052585f616f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.359224 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.359207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzlxw\" (UniqueName: \"kubernetes.io/projected/742c20f3-8c32-4da8-81b2-9052585f616f-kube-api-access-wzlxw\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.459777 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.459754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzlxw\" (UniqueName: \"kubernetes.io/projected/742c20f3-8c32-4da8-81b2-9052585f616f-kube-api-access-wzlxw\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.459890 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.459808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c20f3-8c32-4da8-81b2-9052585f616f-proxy-tls\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.459955 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:32:18.459898    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3e634-predictor-serving-cert: secret "success-200-isvc-3e634-predictor-serving-cert" not found
Sep 17 23:32:18.459955 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:32:18.459940    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/742c20f3-8c32-4da8-81b2-9052585f616f-proxy-tls podName:742c20f3-8c32-4da8-81b2-9052585f616f nodeName:}" failed. No retries permitted until 2026-09-17 23:32:18.959926662 +0000 UTC m=+1301.894372390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/742c20f3-8c32-4da8-81b2-9052585f616f-proxy-tls") pod "success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" (UID: "742c20f3-8c32-4da8-81b2-9052585f616f") : secret "success-200-isvc-3e634-predictor-serving-cert" not found
Sep 17 23:32:18.459955 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.459933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c20f3-8c32-4da8-81b2-9052585f616f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.460576 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.460560    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c20f3-8c32-4da8-81b2-9052585f616f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.468946 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.468924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzlxw\" (UniqueName: \"kubernetes.io/projected/742c20f3-8c32-4da8-81b2-9052585f616f-kube-api-access-wzlxw\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.858738 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.858704    2583 generic.go:356] "Generic (PLEG): container finished" podID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerID="8a839b146d51eaaa96edb7a82929894fe6efc974c158a71a7c4f58bf8ecca3b1" exitCode=2
Sep 17 23:32:18.858887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.858777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" event={"ID":"ccbe12c0-40dc-4b43-884f-1680d643c069","Type":"ContainerDied","Data":"8a839b146d51eaaa96edb7a82929894fe6efc974c158a71a7c4f58bf8ecca3b1"}
Sep 17 23:32:18.963644 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.963617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c20f3-8c32-4da8-81b2-9052585f616f-proxy-tls\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:18.966076 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:18.966050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c20f3-8c32-4da8-81b2-9052585f616f-proxy-tls\") pod \"success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") " pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:19.205972 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:19.205898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:19.482235 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:19.482213    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"]
Sep 17 23:32:19.483919 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:32:19.483894    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742c20f3_8c32_4da8_81b2_9052585f616f.slice/crio-21ea377d83d47d589a1bd07c80ef326678e9a14f5fbb34ffb9e701e4236be7f4 WatchSource:0}: Error finding container 21ea377d83d47d589a1bd07c80ef326678e9a14f5fbb34ffb9e701e4236be7f4: Status 404 returned error can't find the container with id 21ea377d83d47d589a1bd07c80ef326678e9a14f5fbb34ffb9e701e4236be7f4
Sep 17 23:32:19.865886 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:19.865852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" event={"ID":"742c20f3-8c32-4da8-81b2-9052585f616f","Type":"ContainerStarted","Data":"fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003"}
Sep 17 23:32:19.866096 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:19.865918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" event={"ID":"742c20f3-8c32-4da8-81b2-9052585f616f","Type":"ContainerStarted","Data":"71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae"}
Sep 17 23:32:19.866096 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:19.865940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" event={"ID":"742c20f3-8c32-4da8-81b2-9052585f616f","Type":"ContainerStarted","Data":"21ea377d83d47d589a1bd07c80ef326678e9a14f5fbb34ffb9e701e4236be7f4"}
Sep 17 23:32:19.866096 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:19.866057    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:19.866271 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:19.866200    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:19.867422 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:19.867399    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 23:32:19.886119 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:19.886078    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podStartSLOduration=1.886066276 podStartE2EDuration="1.886066276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:19.885486565 +0000 UTC m=+1302.819932316" watchObservedRunningTime="2026-09-17 23:32:19.886066276 +0000 UTC m=+1302.820512084"
Sep 17 23:32:20.326289 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.326247    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 23:32:20.330632 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.330612    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 23:32:20.872042 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.871997    2583 generic.go:356] "Generic (PLEG): container finished" podID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerID="c24b051b7c95b11a1824bfbedccd2afd4fcf0c70d4e30a7eae7dc01bc6286f99" exitCode=0
Sep 17 23:32:20.872164 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.872071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" event={"ID":"ccbe12c0-40dc-4b43-884f-1680d643c069","Type":"ContainerDied","Data":"c24b051b7c95b11a1824bfbedccd2afd4fcf0c70d4e30a7eae7dc01bc6286f99"}
Sep 17 23:32:20.872164 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.872135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk" event={"ID":"ccbe12c0-40dc-4b43-884f-1680d643c069","Type":"ContainerDied","Data":"fac048eccae8b5af47ce382f866ae00623743c629de9109b8c655876f1a0ed60"}
Sep 17 23:32:20.872164 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.872163    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac048eccae8b5af47ce382f866ae00623743c629de9109b8c655876f1a0ed60"
Sep 17 23:32:20.872437 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.872405    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 23:32:20.882607 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.882588    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:32:20.978697 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.978637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbe12c0-40dc-4b43-884f-1680d643c069-proxy-tls\") pod \"ccbe12c0-40dc-4b43-884f-1680d643c069\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") "
Sep 17 23:32:20.978697 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.978694    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-rbac-proxy-sar-config\") pod \"ccbe12c0-40dc-4b43-884f-1680d643c069\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") "
Sep 17 23:32:20.978834 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.978712    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsv4t\" (UniqueName: \"kubernetes.io/projected/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-api-access-hsv4t\") pod \"ccbe12c0-40dc-4b43-884f-1680d643c069\" (UID: \"ccbe12c0-40dc-4b43-884f-1680d643c069\") "
Sep 17 23:32:20.979083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.979052    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ccbe12c0-40dc-4b43-884f-1680d643c069" (UID: "ccbe12c0-40dc-4b43-884f-1680d643c069"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:20.980823 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.980797    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccbe12c0-40dc-4b43-884f-1680d643c069-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ccbe12c0-40dc-4b43-884f-1680d643c069" (UID: "ccbe12c0-40dc-4b43-884f-1680d643c069"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:20.980912 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:20.980852    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-api-access-hsv4t" (OuterVolumeSpecName: "kube-api-access-hsv4t") pod "ccbe12c0-40dc-4b43-884f-1680d643c069" (UID: "ccbe12c0-40dc-4b43-884f-1680d643c069"). InnerVolumeSpecName "kube-api-access-hsv4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:21.079871 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:21.079851    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsv4t\" (UniqueName: \"kubernetes.io/projected/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-api-access-hsv4t\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:32:21.079969 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:21.079873    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccbe12c0-40dc-4b43-884f-1680d643c069-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:32:21.079969 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:21.079883    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccbe12c0-40dc-4b43-884f-1680d643c069-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:32:21.601980 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:21.601943    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:32:21.876364 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:21.876287    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"
Sep 17 23:32:21.911458 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:21.911425    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"]
Sep 17 23:32:21.929281 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:21.929243    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-11096-predictor-587bc899cb-dtvwk"]
Sep 17 23:32:23.157345 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:23.157304    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" path="/var/lib/kubelet/pods/ccbe12c0-40dc-4b43-884f-1680d643c069/volumes"
Sep 17 23:32:25.884201 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:25.884170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:32:25.884615 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:25.884584    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 23:32:26.601206 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:26.601168    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:32:31.602407 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:31.602362    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:32:31.602782 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:31.602483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:32:35.884761 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:35.884716    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 23:32:36.602040 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:36.601988    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:32:39.184804 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:39.184775    2583 scope.go:117] "RemoveContainer" containerID="8a839b146d51eaaa96edb7a82929894fe6efc974c158a71a7c4f58bf8ecca3b1"
Sep 17 23:32:39.193741 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:39.193721    2583 scope.go:117] "RemoveContainer" containerID="c24b051b7c95b11a1824bfbedccd2afd4fcf0c70d4e30a7eae7dc01bc6286f99"
Sep 17 23:32:41.601438 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:41.601400    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:32:45.885014 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:45.884968    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 23:32:46.601456 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:46.601416    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:32:48.163183 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.163159    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:32:48.287785 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.287758    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-openshift-service-ca-bundle\") pod \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\" (UID: \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\") "
Sep 17 23:32:48.287961 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.287818    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-proxy-tls\") pod \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\" (UID: \"9ae51a90-c525-49e0-a1f8-d2c95f7194b9\") "
Sep 17 23:32:48.288166 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.288143    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9ae51a90-c525-49e0-a1f8-d2c95f7194b9" (UID: "9ae51a90-c525-49e0-a1f8-d2c95f7194b9"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:48.289844 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.289824    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ae51a90-c525-49e0-a1f8-d2c95f7194b9" (UID: "9ae51a90-c525-49e0-a1f8-d2c95f7194b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:48.388977 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.388942    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:32:48.388977 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.388970    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae51a90-c525-49e0-a1f8-d2c95f7194b9-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:32:48.998011 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.997975    2583 generic.go:356] "Generic (PLEG): container finished" podID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerID="a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f" exitCode=0
Sep 17 23:32:48.998311 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.998016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" event={"ID":"9ae51a90-c525-49e0-a1f8-d2c95f7194b9","Type":"ContainerDied","Data":"a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f"}
Sep 17 23:32:48.998311 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.998096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw" event={"ID":"9ae51a90-c525-49e0-a1f8-d2c95f7194b9","Type":"ContainerDied","Data":"601e9c8ae297d13c5268c74b648eeba7a94900f196291bdba42b0b6fecf55565"}
Sep 17 23:32:48.998311 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.998119    2583 scope.go:117] "RemoveContainer" containerID="a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f"
Sep 17 23:32:48.998311 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:48.998044    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"
Sep 17 23:32:49.006979 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:49.006964    2583 scope.go:117] "RemoveContainer" containerID="a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f"
Sep 17 23:32:49.007248 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:32:49.007228    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f\": container with ID starting with a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f not found: ID does not exist" containerID="a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f"
Sep 17 23:32:49.007301 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:49.007258    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f"} err="failed to get container status \"a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f\": rpc error: code = NotFound desc = could not find container \"a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f\": container with ID starting with a7c3baaadcc448ce598087449d7e3ffa7c3a4da648164782f0ac38b0918a7f2f not found: ID does not exist"
Sep 17 23:32:49.018895 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:49.018866    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"]
Sep 17 23:32:49.027189 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:49.027163    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-11096-d974cf6bd-77wlw"]
Sep 17 23:32:49.157894 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:49.157850    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" path="/var/lib/kubelet/pods/9ae51a90-c525-49e0-a1f8-d2c95f7194b9/volumes"
Sep 17 23:32:55.885359 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:32:55.885273    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 23:33:04.457995 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.457962    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"]
Sep 17 23:33:04.458443 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.458224    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c" containerID="cri-o://5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034" gracePeriod=30
Sep 17 23:33:04.560592 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.560562    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"]
Sep 17 23:33:04.560812 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.560783    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container" containerID="cri-o://394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968" gracePeriod=30
Sep 17 23:33:04.560919 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.560852    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kube-rbac-proxy" containerID="cri-o://ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b" gracePeriod=30
Sep 17 23:33:04.702320 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702288    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"]
Sep 17 23:33:04.702703 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702684    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container"
Sep 17 23:33:04.702786 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702705    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container"
Sep 17 23:33:04.702786 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702727    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kube-rbac-proxy"
Sep 17 23:33:04.702786 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702736    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kube-rbac-proxy"
Sep 17 23:33:04.702786 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702761    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096"
Sep 17 23:33:04.702786 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702770    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096"
Sep 17 23:33:04.703054 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702883    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kube-rbac-proxy"
Sep 17 23:33:04.703054 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702898    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae51a90-c525-49e0-a1f8-d2c95f7194b9" containerName="switch-graph-11096"
Sep 17 23:33:04.703054 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.702910    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbe12c0-40dc-4b43-884f-1680d643c069" containerName="kserve-container"
Sep 17 23:33:04.705982 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.705960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:04.709314 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.709270    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4a1f9-kube-rbac-proxy-sar-config\""
Sep 17 23:33:04.709724 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.709709    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4a1f9-predictor-serving-cert\""
Sep 17 23:33:04.720737 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.720718    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"]
Sep 17 23:33:04.818599 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.818571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39644fba-18d3-471e-ac91-efe1b9de34df-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") " pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:04.818720 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.818602    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39644fba-18d3-471e-ac91-efe1b9de34df-proxy-tls\") pod \"success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") " pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:04.818720 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.818628    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfr2\" (UniqueName: \"kubernetes.io/projected/39644fba-18d3-471e-ac91-efe1b9de34df-kube-api-access-7pfr2\") pod \"success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") " pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:04.919483 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.919458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39644fba-18d3-471e-ac91-efe1b9de34df-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") " pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:04.919604 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.919489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39644fba-18d3-471e-ac91-efe1b9de34df-proxy-tls\") pod \"success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") " pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:04.919604 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.919511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfr2\" (UniqueName: \"kubernetes.io/projected/39644fba-18d3-471e-ac91-efe1b9de34df-kube-api-access-7pfr2\") pod \"success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") " pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:04.920352 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.920327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39644fba-18d3-471e-ac91-efe1b9de34df-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") " pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:04.921994 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.921973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39644fba-18d3-471e-ac91-efe1b9de34df-proxy-tls\") pod \"success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") " pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:04.928410 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:04.928388    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfr2\" (UniqueName: \"kubernetes.io/projected/39644fba-18d3-471e-ac91-efe1b9de34df-kube-api-access-7pfr2\") pod \"success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") " pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:05.015342 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:05.015291    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:05.072149 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:05.072116    2583 generic.go:356] "Generic (PLEG): container finished" podID="20639def-b323-4067-ade5-b36de99fa233" containerID="ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b" exitCode=2
Sep 17 23:33:05.072276 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:05.072168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" event={"ID":"20639def-b323-4067-ade5-b36de99fa233","Type":"ContainerDied","Data":"ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b"}
Sep 17 23:33:05.289925 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:05.289902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"]
Sep 17 23:33:05.291540 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:33:05.291515    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39644fba_18d3_471e_ac91_efe1b9de34df.slice/crio-1d79aa5c612f1c1a84ce9f2f9cf6427f0c575c5d7df4336ace33b5ca52e937d1 WatchSource:0}: Error finding container 1d79aa5c612f1c1a84ce9f2f9cf6427f0c575c5d7df4336ace33b5ca52e937d1: Status 404 returned error can't find the container with id 1d79aa5c612f1c1a84ce9f2f9cf6427f0c575c5d7df4336ace33b5ca52e937d1
Sep 17 23:33:05.886083 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:05.886053    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:33:06.077937 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:06.077900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" event={"ID":"39644fba-18d3-471e-ac91-efe1b9de34df","Type":"ContainerStarted","Data":"56157cbd0b9dcc451eff1c8c1e584d37c5887ded7773903b50c6c03861c34679"}
Sep 17 23:33:06.078103 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:06.077961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" event={"ID":"39644fba-18d3-471e-ac91-efe1b9de34df","Type":"ContainerStarted","Data":"eb177c7eaf41c654129e355fa4fc3df0180132985da161fda959849e6dff17f5"}
Sep 17 23:33:06.078103 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:06.077988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" event={"ID":"39644fba-18d3-471e-ac91-efe1b9de34df","Type":"ContainerStarted","Data":"1d79aa5c612f1c1a84ce9f2f9cf6427f0c575c5d7df4336ace33b5ca52e937d1"}
Sep 17 23:33:06.078103 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:06.078014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:06.097556 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:06.097506    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" podStartSLOduration=2.097487622 podStartE2EDuration="2.097487622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:06.096061932 +0000 UTC m=+1349.030507682" watchObservedRunningTime="2026-09-17 23:33:06.097487622 +0000 UTC m=+1349.031933375"
Sep 17 23:33:06.801599 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:06.801562    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:07.088417 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.088333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:07.089807 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.089776    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:33:07.410623 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.410602    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:33:07.539229 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.539202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20639def-b323-4067-ade5-b36de99fa233-kube-rbac-proxy-sar-config\") pod \"20639def-b323-4067-ade5-b36de99fa233\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") "
Sep 17 23:33:07.539364 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.539257    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpsps\" (UniqueName: \"kubernetes.io/projected/20639def-b323-4067-ade5-b36de99fa233-kube-api-access-cpsps\") pod \"20639def-b323-4067-ade5-b36de99fa233\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") "
Sep 17 23:33:07.539364 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.539289    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls\") pod \"20639def-b323-4067-ade5-b36de99fa233\" (UID: \"20639def-b323-4067-ade5-b36de99fa233\") "
Sep 17 23:33:07.539606 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.539582    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20639def-b323-4067-ade5-b36de99fa233-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20639def-b323-4067-ade5-b36de99fa233" (UID: "20639def-b323-4067-ade5-b36de99fa233"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:07.541445 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.541420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20639def-b323-4067-ade5-b36de99fa233" (UID: "20639def-b323-4067-ade5-b36de99fa233"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:07.541533 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.541437    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20639def-b323-4067-ade5-b36de99fa233-kube-api-access-cpsps" (OuterVolumeSpecName: "kube-api-access-cpsps") pod "20639def-b323-4067-ade5-b36de99fa233" (UID: "20639def-b323-4067-ade5-b36de99fa233"). InnerVolumeSpecName "kube-api-access-cpsps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:07.640410 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.640385    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20639def-b323-4067-ade5-b36de99fa233-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:07.640410 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.640407    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20639def-b323-4067-ade5-b36de99fa233-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:07.640531 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:07.640416    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpsps\" (UniqueName: \"kubernetes.io/projected/20639def-b323-4067-ade5-b36de99fa233-kube-api-access-cpsps\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:08.094169 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.094131    2583 generic.go:356] "Generic (PLEG): container finished" podID="20639def-b323-4067-ade5-b36de99fa233" containerID="394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968" exitCode=0
Sep 17 23:33:08.094523 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.094181    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"
Sep 17 23:33:08.094523 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.094211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" event={"ID":"20639def-b323-4067-ade5-b36de99fa233","Type":"ContainerDied","Data":"394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968"}
Sep 17 23:33:08.094523 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.094258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv" event={"ID":"20639def-b323-4067-ade5-b36de99fa233","Type":"ContainerDied","Data":"723dab41925e0066b48407047dabcc021bc7c979f098140c87bfc8ab2835cf70"}
Sep 17 23:33:08.094523 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.094286    2583 scope.go:117] "RemoveContainer" containerID="ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b"
Sep 17 23:33:08.094845 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.094816    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:33:08.103698 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.103680    2583 scope.go:117] "RemoveContainer" containerID="394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968"
Sep 17 23:33:08.110887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.110869    2583 scope.go:117] "RemoveContainer" containerID="ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b"
Sep 17 23:33:08.111262 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:08.111241    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b\": container with ID starting with ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b not found: ID does not exist" containerID="ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b"
Sep 17 23:33:08.111331 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.111268    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b"} err="failed to get container status \"ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b\": rpc error: code = NotFound desc = could not find container \"ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b\": container with ID starting with ced61c74dbd1c676e2d0137239e7d1ea5e825ca6a03449742dfe4c4461b9c46b not found: ID does not exist"
Sep 17 23:33:08.111331 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.111284    2583 scope.go:117] "RemoveContainer" containerID="394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968"
Sep 17 23:33:08.111555 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:08.111535    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968\": container with ID starting with 394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968 not found: ID does not exist" containerID="394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968"
Sep 17 23:33:08.111597 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.111561    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968"} err="failed to get container status \"394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968\": rpc error: code = NotFound desc = could not find container \"394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968\": container with ID starting with 394eeb46d4317c0b6fcb8ee66e6db9b65197b168194280a0ce18bba022b83968 not found: ID does not exist"
Sep 17 23:33:08.116920 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.116896    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"]
Sep 17 23:33:08.121080 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:08.121058    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2026c-predictor-866c975b8d-dr7dv"]
Sep 17 23:33:09.156079 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:09.156050    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20639def-b323-4067-ade5-b36de99fa233" path="/var/lib/kubelet/pods/20639def-b323-4067-ade5-b36de99fa233/volumes"
Sep 17 23:33:11.802152 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:11.802103    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:13.099379 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:13.099352    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:13.099854 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:13.099826    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:33:16.801867 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:16.801825    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:16.802241 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:16.801933    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:33:18.257765 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.257732    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"]
Sep 17 23:33:18.258154 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.258126    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container"
Sep 17 23:33:18.258154 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.258139    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container"
Sep 17 23:33:18.258258 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.258162    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kube-rbac-proxy"
Sep 17 23:33:18.258258 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.258168    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kube-rbac-proxy"
Sep 17 23:33:18.258258 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.258232    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kserve-container"
Sep 17 23:33:18.258258 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.258242    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20639def-b323-4067-ade5-b36de99fa233" containerName="kube-rbac-proxy"
Sep 17 23:33:18.262836 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.262814    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:18.265050 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.265010    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-3e634-serving-cert\""
Sep 17 23:33:18.265144 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.265094    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-3e634-kube-rbac-proxy-sar-config\""
Sep 17 23:33:18.275502 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.275481    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"]
Sep 17 23:33:18.416048 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.415989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-proxy-tls\") pod \"ensemble-graph-3e634-5f7d5cf85c-d6nd4\" (UID: \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\") " pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:18.416218 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.416062    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-openshift-service-ca-bundle\") pod \"ensemble-graph-3e634-5f7d5cf85c-d6nd4\" (UID: \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\") " pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:18.516929 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.516852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-openshift-service-ca-bundle\") pod \"ensemble-graph-3e634-5f7d5cf85c-d6nd4\" (UID: \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\") " pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:18.517101 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.516959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-proxy-tls\") pod \"ensemble-graph-3e634-5f7d5cf85c-d6nd4\" (UID: \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\") " pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:18.517531 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.517510    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-openshift-service-ca-bundle\") pod \"ensemble-graph-3e634-5f7d5cf85c-d6nd4\" (UID: \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\") " pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:18.519290 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.519269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-proxy-tls\") pod \"ensemble-graph-3e634-5f7d5cf85c-d6nd4\" (UID: \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\") " pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:18.573272 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.573248    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:18.845627 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:18.845603    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"]
Sep 17 23:33:18.847629 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:33:18.847592    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cffc0f5_ef77_4e6d_a972_dd18eb131c6d.slice/crio-6a13e125bd1a4cc33d2a3e464c6205de0884000cdff1f242f1d10f1bb86c5de0 WatchSource:0}: Error finding container 6a13e125bd1a4cc33d2a3e464c6205de0884000cdff1f242f1d10f1bb86c5de0: Status 404 returned error can't find the container with id 6a13e125bd1a4cc33d2a3e464c6205de0884000cdff1f242f1d10f1bb86c5de0
Sep 17 23:33:19.144984 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:19.144900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" event={"ID":"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d","Type":"ContainerStarted","Data":"ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb"}
Sep 17 23:33:19.144984 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:19.144949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" event={"ID":"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d","Type":"ContainerStarted","Data":"6a13e125bd1a4cc33d2a3e464c6205de0884000cdff1f242f1d10f1bb86c5de0"}
Sep 17 23:33:19.144984 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:19.144981    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:19.162294 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:19.162245    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" podStartSLOduration=1.162230454 podStartE2EDuration="1.162230454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:19.160855307 +0000 UTC m=+1362.095301040" watchObservedRunningTime="2026-09-17 23:33:19.162230454 +0000 UTC m=+1362.096676229"
Sep 17 23:33:21.801395 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:21.801355    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:23.100753 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:23.100714    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:33:25.156734 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:25.156709    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:26.801522 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:26.801484    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:28.326981 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.326949    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"]
Sep 17 23:33:28.327449 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.327161    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634" containerID="cri-o://ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb" gracePeriod=30
Sep 17 23:33:28.438807 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.438779    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"]
Sep 17 23:33:28.438995 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.438970    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container" containerID="cri-o://71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae" gracePeriod=30
Sep 17 23:33:28.439147 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.439008    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kube-rbac-proxy" containerID="cri-o://fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003" gracePeriod=30
Sep 17 23:33:28.567777 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.567748    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"]
Sep 17 23:33:28.571203 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.571183    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:28.573639 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.573619    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d6263-kube-rbac-proxy-sar-config\""
Sep 17 23:33:28.573749 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.573688    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d6263-predictor-serving-cert\""
Sep 17 23:33:28.584670 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.584651    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"]
Sep 17 23:33:28.687591 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.687564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31174a96-f041-4e06-878b-0e4487e22ca3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:28.687730 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.687597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:28.687730 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.687625    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4l7\" (UniqueName: \"kubernetes.io/projected/31174a96-f041-4e06-878b-0e4487e22ca3-kube-api-access-td4l7\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:28.788407 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.788378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31174a96-f041-4e06-878b-0e4487e22ca3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:28.788407 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.788409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:28.788569 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.788436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4l7\" (UniqueName: \"kubernetes.io/projected/31174a96-f041-4e06-878b-0e4487e22ca3-kube-api-access-td4l7\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:28.788569 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:28.788537    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d6263-predictor-serving-cert: secret "success-200-isvc-d6263-predictor-serving-cert" not found
Sep 17 23:33:28.788663 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:28.788602    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls podName:31174a96-f041-4e06-878b-0e4487e22ca3 nodeName:}" failed. No retries permitted until 2026-09-17 23:33:29.288586479 +0000 UTC m=+1372.223032212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls") pod "success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" (UID: "31174a96-f041-4e06-878b-0e4487e22ca3") : secret "success-200-isvc-d6263-predictor-serving-cert" not found
Sep 17 23:33:28.789177 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.789162    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31174a96-f041-4e06-878b-0e4487e22ca3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:28.800217 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:28.800196    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4l7\" (UniqueName: \"kubernetes.io/projected/31174a96-f041-4e06-878b-0e4487e22ca3-kube-api-access-td4l7\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:29.188630 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:29.188595    2583 generic.go:356] "Generic (PLEG): container finished" podID="742c20f3-8c32-4da8-81b2-9052585f616f" containerID="fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003" exitCode=2
Sep 17 23:33:29.188801 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:29.188649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" event={"ID":"742c20f3-8c32-4da8-81b2-9052585f616f","Type":"ContainerDied","Data":"fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003"}
Sep 17 23:33:29.292168 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:29.292130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:29.292334 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:29.292288    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d6263-predictor-serving-cert: secret "success-200-isvc-d6263-predictor-serving-cert" not found
Sep 17 23:33:29.292382 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:29.292371    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls podName:31174a96-f041-4e06-878b-0e4487e22ca3 nodeName:}" failed. No retries permitted until 2026-09-17 23:33:30.292355609 +0000 UTC m=+1373.226801343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls") pod "success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" (UID: "31174a96-f041-4e06-878b-0e4487e22ca3") : secret "success-200-isvc-d6263-predictor-serving-cert" not found
Sep 17 23:33:30.154116 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:30.154081    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:30.299912 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:30.299863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:30.302336 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:30.302317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls\") pod \"success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") " pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:30.381413 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:30.381380    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:30.652287 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:30.652232    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"]
Sep 17 23:33:30.654494 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:33:30.654461    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31174a96_f041_4e06_878b_0e4487e22ca3.slice/crio-351121b195f29295333cd1edc7f33e7588b9ec2e7f127d3fa9ef2b4131c70b9a WatchSource:0}: Error finding container 351121b195f29295333cd1edc7f33e7588b9ec2e7f127d3fa9ef2b4131c70b9a: Status 404 returned error can't find the container with id 351121b195f29295333cd1edc7f33e7588b9ec2e7f127d3fa9ef2b4131c70b9a
Sep 17 23:33:30.873431 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:30.873386    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 17 23:33:31.205411 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.205374    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" event={"ID":"31174a96-f041-4e06-878b-0e4487e22ca3","Type":"ContainerStarted","Data":"6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590"}
Sep 17 23:33:31.205864 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.205457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" event={"ID":"31174a96-f041-4e06-878b-0e4487e22ca3","Type":"ContainerStarted","Data":"77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7"}
Sep 17 23:33:31.205864 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.205480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" event={"ID":"31174a96-f041-4e06-878b-0e4487e22ca3","Type":"ContainerStarted","Data":"351121b195f29295333cd1edc7f33e7588b9ec2e7f127d3fa9ef2b4131c70b9a"}
Sep 17 23:33:31.205864 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.205507    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:31.224992 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.224933    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" podStartSLOduration=3.224911496 podStartE2EDuration="3.224911496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:31.222353765 +0000 UTC m=+1374.156799540" watchObservedRunningTime="2026-09-17 23:33:31.224911496 +0000 UTC m=+1374.159357247"
Sep 17 23:33:31.488624 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.488595    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:33:31.610412 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.610379    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzlxw\" (UniqueName: \"kubernetes.io/projected/742c20f3-8c32-4da8-81b2-9052585f616f-kube-api-access-wzlxw\") pod \"742c20f3-8c32-4da8-81b2-9052585f616f\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") "
Sep 17 23:33:31.610587 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.610433    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c20f3-8c32-4da8-81b2-9052585f616f-proxy-tls\") pod \"742c20f3-8c32-4da8-81b2-9052585f616f\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") "
Sep 17 23:33:31.610587 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.610549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c20f3-8c32-4da8-81b2-9052585f616f-kube-rbac-proxy-sar-config\") pod \"742c20f3-8c32-4da8-81b2-9052585f616f\" (UID: \"742c20f3-8c32-4da8-81b2-9052585f616f\") "
Sep 17 23:33:31.610943 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.610912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742c20f3-8c32-4da8-81b2-9052585f616f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "742c20f3-8c32-4da8-81b2-9052585f616f" (UID: "742c20f3-8c32-4da8-81b2-9052585f616f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:31.612564 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.612532    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742c20f3-8c32-4da8-81b2-9052585f616f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "742c20f3-8c32-4da8-81b2-9052585f616f" (UID: "742c20f3-8c32-4da8-81b2-9052585f616f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:31.612657 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.612571    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742c20f3-8c32-4da8-81b2-9052585f616f-kube-api-access-wzlxw" (OuterVolumeSpecName: "kube-api-access-wzlxw") pod "742c20f3-8c32-4da8-81b2-9052585f616f" (UID: "742c20f3-8c32-4da8-81b2-9052585f616f"). InnerVolumeSpecName "kube-api-access-wzlxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:31.712150 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.712119    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzlxw\" (UniqueName: \"kubernetes.io/projected/742c20f3-8c32-4da8-81b2-9052585f616f-kube-api-access-wzlxw\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:31.712150 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.712145    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742c20f3-8c32-4da8-81b2-9052585f616f-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:31.712150 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.712157    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742c20f3-8c32-4da8-81b2-9052585f616f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:31.801473 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:31.801440    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:32.210986 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.210898    2583 generic.go:356] "Generic (PLEG): container finished" podID="742c20f3-8c32-4da8-81b2-9052585f616f" containerID="71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae" exitCode=0
Sep 17 23:33:32.210986 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.210947    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"
Sep 17 23:33:32.210986 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.210966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" event={"ID":"742c20f3-8c32-4da8-81b2-9052585f616f","Type":"ContainerDied","Data":"71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae"}
Sep 17 23:33:32.211506 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.211051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s" event={"ID":"742c20f3-8c32-4da8-81b2-9052585f616f","Type":"ContainerDied","Data":"21ea377d83d47d589a1bd07c80ef326678e9a14f5fbb34ffb9e701e4236be7f4"}
Sep 17 23:33:32.211506 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.211066    2583 scope.go:117] "RemoveContainer" containerID="fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003"
Sep 17 23:33:32.211506 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.211401    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:32.212916 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.212887    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:33:32.219901 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.219854    2583 scope.go:117] "RemoveContainer" containerID="71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae"
Sep 17 23:33:32.227731 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.227713    2583 scope.go:117] "RemoveContainer" containerID="fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003"
Sep 17 23:33:32.227977 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:32.227962    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003\": container with ID starting with fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003 not found: ID does not exist" containerID="fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003"
Sep 17 23:33:32.228041 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.227985    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003"} err="failed to get container status \"fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003\": rpc error: code = NotFound desc = could not find container \"fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003\": container with ID starting with fb7a67a4abf22d32a21c4007f2cde9103c05b88900d737fbf303fb8e54703003 not found: ID does not exist"
Sep 17 23:33:32.228041 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.228000    2583 scope.go:117] "RemoveContainer" containerID="71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae"
Sep 17 23:33:32.228261 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:32.228247    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae\": container with ID starting with 71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae not found: ID does not exist" containerID="71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae"
Sep 17 23:33:32.228301 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.228263    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae"} err="failed to get container status \"71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae\": rpc error: code = NotFound desc = could not find container \"71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae\": container with ID starting with 71b9bdc5f36c80aaa42e327dbc3318a52d61d35518732c90996c62794c06daae not found: ID does not exist"
Sep 17 23:33:32.233786 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.233762    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"]
Sep 17 23:33:32.237414 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:32.237395    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e634-predictor-6bd488c9d4-k5j9s"]
Sep 17 23:33:33.099869 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:33.099831    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:33:33.157119 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:33.157087    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" path="/var/lib/kubelet/pods/742c20f3-8c32-4da8-81b2-9052585f616f/volumes"
Sep 17 23:33:33.216471 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:33.216433    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:33:35.107381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.107358    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:33:35.139259 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.139229    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-proxy-tls\") pod \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\" (UID: \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\") "
Sep 17 23:33:35.139429 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.139296    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-openshift-service-ca-bundle\") pod \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\" (UID: \"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e\") "
Sep 17 23:33:35.139629 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.139607    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" (UID: "5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:35.141290 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.141267    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" (UID: "5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:35.154403 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.154370    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:35.228055 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.227954    2583 generic.go:356] "Generic (PLEG): container finished" podID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerID="5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034" exitCode=0
Sep 17 23:33:35.228055 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.227999    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"
Sep 17 23:33:35.228055 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.228003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" event={"ID":"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e","Type":"ContainerDied","Data":"5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034"}
Sep 17 23:33:35.228324 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.228065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj" event={"ID":"5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e","Type":"ContainerDied","Data":"ae5b6e767b5ae5660d153ded346d1ce95491fb939faad4e3acee2e47914bb16d"}
Sep 17 23:33:35.228324 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.228088    2583 scope.go:117] "RemoveContainer" containerID="5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034"
Sep 17 23:33:35.236241 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.236222    2583 scope.go:117] "RemoveContainer" containerID="5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034"
Sep 17 23:33:35.236507 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:35.236487    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034\": container with ID starting with 5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034 not found: ID does not exist" containerID="5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034"
Sep 17 23:33:35.236552 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.236514    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034"} err="failed to get container status \"5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034\": rpc error: code = NotFound desc = could not find container \"5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034\": container with ID starting with 5f9364af45afc677b99515f589199db7de6760cf4a9b83511ea8d7b48af13034 not found: ID does not exist"
Sep 17 23:33:35.239864 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.239844    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:35.239945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.239867    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:35.244236 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.244214    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"]
Sep 17 23:33:35.248189 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:35.248170    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2026c-6498f87569-5x5bj"]
Sep 17 23:33:37.157000 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:37.156964    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" path="/var/lib/kubelet/pods/5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e/volumes"
Sep 17 23:33:38.221653 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:38.221624    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:33:38.222202 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:38.222171    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:33:40.153387 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:40.153346    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:40.153738 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:40.153459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:43.100540 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:43.100502    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 23:33:45.153248 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:45.153209    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:48.222777 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:48.222738    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:33:50.153612 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:50.153567    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:53.100367 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:53.100332    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:33:55.153626 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:55.153574    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:33:58.222226 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:58.222184    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:33:58.472392 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:58.472344    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:58.481121 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:58.481099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-openshift-service-ca-bundle\") pod \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\" (UID: \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\") "
Sep 17 23:33:58.481202 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:58.481172    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-proxy-tls\") pod \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\" (UID: \"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d\") "
Sep 17 23:33:58.481440 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:58.481416    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" (UID: "8cffc0f5-ef77-4e6d-a972-dd18eb131c6d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:58.483136 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:58.483113    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" (UID: "8cffc0f5-ef77-4e6d-a972-dd18eb131c6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:58.581856 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:58.581836    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:58.581856 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:58.581855    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:33:59.345879 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:59.345850    2583 generic.go:356] "Generic (PLEG): container finished" podID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerID="ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb" exitCode=0
Sep 17 23:33:59.346314 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:59.345895    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"
Sep 17 23:33:59.346314 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:59.345929    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" event={"ID":"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d","Type":"ContainerDied","Data":"ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb"}
Sep 17 23:33:59.346314 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:59.345979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4" event={"ID":"8cffc0f5-ef77-4e6d-a972-dd18eb131c6d","Type":"ContainerDied","Data":"6a13e125bd1a4cc33d2a3e464c6205de0884000cdff1f242f1d10f1bb86c5de0"}
Sep 17 23:33:59.346314 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:59.346002    2583 scope.go:117] "RemoveContainer" containerID="ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb"
Sep 17 23:33:59.354217 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:59.354199    2583 scope.go:117] "RemoveContainer" containerID="ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb"
Sep 17 23:33:59.354474 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:33:59.354454    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb\": container with ID starting with ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb not found: ID does not exist" containerID="ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb"
Sep 17 23:33:59.354521 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:59.354481    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb"} err="failed to get container status \"ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb\": rpc error: code = NotFound desc = could not find container \"ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb\": container with ID starting with ec252982c86a5922fce533f0f12ffddc87cffff40a0382aa7fb5aee198d339cb not found: ID does not exist"
Sep 17 23:33:59.365463 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:59.365442    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"]
Sep 17 23:33:59.370490 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:33:59.370471    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-3e634-5f7d5cf85c-d6nd4"]
Sep 17 23:34:01.156318 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:01.156288    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" path="/var/lib/kubelet/pods/8cffc0f5-ef77-4e6d-a972-dd18eb131c6d/volumes"
Sep 17 23:34:04.679701 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.679668    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"]
Sep 17 23:34:04.680104 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680081    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kube-rbac-proxy"
Sep 17 23:34:04.680104 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680099    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kube-rbac-proxy"
Sep 17 23:34:04.680181 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680110    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container"
Sep 17 23:34:04.680181 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680115    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container"
Sep 17 23:34:04.680181 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680126    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c"
Sep 17 23:34:04.680181 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680131    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c"
Sep 17 23:34:04.680181 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680139    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634"
Sep 17 23:34:04.680181 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680144    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634"
Sep 17 23:34:04.680357 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680209    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f608d0b-e29e-4f6f-b8e0-edb51c43fa7e" containerName="sequence-graph-2026c"
Sep 17 23:34:04.680357 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680220    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cffc0f5-ef77-4e6d-a972-dd18eb131c6d" containerName="ensemble-graph-3e634"
Sep 17 23:34:04.680357 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680227    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kube-rbac-proxy"
Sep 17 23:34:04.680357 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.680237    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="742c20f3-8c32-4da8-81b2-9052585f616f" containerName="kserve-container"
Sep 17 23:34:04.684538 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.684520    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:04.686827 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.686803    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4a1f9-kube-rbac-proxy-sar-config\""
Sep 17 23:34:04.686920 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.686860    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4a1f9-serving-cert\""
Sep 17 23:34:04.691807 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.691787    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"]
Sep 17 23:34:04.723628 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.723603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls\") pod \"sequence-graph-4a1f9-5868fb9865-dwz95\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") " pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:04.723717 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.723642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-openshift-service-ca-bundle\") pod \"sequence-graph-4a1f9-5868fb9865-dwz95\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") " pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:04.824198 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.824156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls\") pod \"sequence-graph-4a1f9-5868fb9865-dwz95\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") " pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:04.824349 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.824210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-openshift-service-ca-bundle\") pod \"sequence-graph-4a1f9-5868fb9865-dwz95\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") " pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:04.824708 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:34:04.824553    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-4a1f9-serving-cert: secret "sequence-graph-4a1f9-serving-cert" not found
Sep 17 23:34:04.824708 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:34:04.824696    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls podName:2fb527d6-dff9-4e49-b3ff-242bd1db8fb8 nodeName:}" failed. No retries permitted until 2026-09-17 23:34:05.324674581 +0000 UTC m=+1408.259120324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls") pod "sequence-graph-4a1f9-5868fb9865-dwz95" (UID: "2fb527d6-dff9-4e49-b3ff-242bd1db8fb8") : secret "sequence-graph-4a1f9-serving-cert" not found
Sep 17 23:34:04.824893 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:04.824863    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-openshift-service-ca-bundle\") pod \"sequence-graph-4a1f9-5868fb9865-dwz95\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") " pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:05.327223 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:05.327194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls\") pod \"sequence-graph-4a1f9-5868fb9865-dwz95\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") " pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:05.327395 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:34:05.327302    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-4a1f9-serving-cert: secret "sequence-graph-4a1f9-serving-cert" not found
Sep 17 23:34:05.327395 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:34:05.327377    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls podName:2fb527d6-dff9-4e49-b3ff-242bd1db8fb8 nodeName:}" failed. No retries permitted until 2026-09-17 23:34:06.327356543 +0000 UTC m=+1409.261802290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls") pod "sequence-graph-4a1f9-5868fb9865-dwz95" (UID: "2fb527d6-dff9-4e49-b3ff-242bd1db8fb8") : secret "sequence-graph-4a1f9-serving-cert" not found
Sep 17 23:34:06.337617 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:06.337582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls\") pod \"sequence-graph-4a1f9-5868fb9865-dwz95\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") " pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:06.339945 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:06.339917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls\") pod \"sequence-graph-4a1f9-5868fb9865-dwz95\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") " pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:06.495029 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:06.495002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:06.765879 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:06.765854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"]
Sep 17 23:34:06.767705 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:34:06.767667    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb527d6_dff9_4e49_b3ff_242bd1db8fb8.slice/crio-8e2bf1d4698b338d3de09b457e51a6e594c7ddc0704a90f5c47a75c853a63936 WatchSource:0}: Error finding container 8e2bf1d4698b338d3de09b457e51a6e594c7ddc0704a90f5c47a75c853a63936: Status 404 returned error can't find the container with id 8e2bf1d4698b338d3de09b457e51a6e594c7ddc0704a90f5c47a75c853a63936
Sep 17 23:34:07.382664 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:07.382635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" event={"ID":"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8","Type":"ContainerStarted","Data":"10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d"}
Sep 17 23:34:07.383004 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:07.382679    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" event={"ID":"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8","Type":"ContainerStarted","Data":"8e2bf1d4698b338d3de09b457e51a6e594c7ddc0704a90f5c47a75c853a63936"}
Sep 17 23:34:07.383004 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:07.382702    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:07.400383 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:07.400337    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" podStartSLOduration=3.400322241 podStartE2EDuration="3.400322241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:07.398986581 +0000 UTC m=+1410.333432332" watchObservedRunningTime="2026-09-17 23:34:07.400322241 +0000 UTC m=+1410.334767991"
Sep 17 23:34:08.222413 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:08.222377    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 23:34:13.393175 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:13.393144    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:14.759296 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:14.759266    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"]
Sep 17 23:34:14.759732 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:14.759467    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9" containerID="cri-o://10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d" gracePeriod=30
Sep 17 23:34:14.871714 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:14.871680    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"]
Sep 17 23:34:14.871961 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:14.871936    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container" containerID="cri-o://eb177c7eaf41c654129e355fa4fc3df0180132985da161fda959849e6dff17f5" gracePeriod=30
Sep 17 23:34:14.872063 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:14.871996    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kube-rbac-proxy" containerID="cri-o://56157cbd0b9dcc451eff1c8c1e584d37c5887ded7773903b50c6c03861c34679" gracePeriod=30
Sep 17 23:34:14.997966 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:14.997936    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"]
Sep 17 23:34:15.001174 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.001156    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.003662 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.003643    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-51963-predictor-serving-cert\""
Sep 17 23:34:15.004409 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.004392    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-51963-kube-rbac-proxy-sar-config\""
Sep 17 23:34:15.013233 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.013188    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"]
Sep 17 23:34:15.103369 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.103344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835d0960-08b8-47c9-bf25-8b4df6b73001-proxy-tls\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.103495 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.103394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.103495 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.103449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbzk\" (UniqueName: \"kubernetes.io/projected/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-api-access-pdbzk\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.203900 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.203871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.204004 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.203906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbzk\" (UniqueName: \"kubernetes.io/projected/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-api-access-pdbzk\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.204004 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.203947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835d0960-08b8-47c9-bf25-8b4df6b73001-proxy-tls\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.204120 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:34:15.204055    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-51963-predictor-serving-cert: secret "success-200-isvc-51963-predictor-serving-cert" not found
Sep 17 23:34:15.204120 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:34:15.204111    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835d0960-08b8-47c9-bf25-8b4df6b73001-proxy-tls podName:835d0960-08b8-47c9-bf25-8b4df6b73001 nodeName:}" failed. No retries permitted until 2026-09-17 23:34:15.704096726 +0000 UTC m=+1418.638542455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/835d0960-08b8-47c9-bf25-8b4df6b73001-proxy-tls") pod "success-200-isvc-51963-predictor-55b958c4f8-6sthm" (UID: "835d0960-08b8-47c9-bf25-8b4df6b73001") : secret "success-200-isvc-51963-predictor-serving-cert" not found
Sep 17 23:34:15.204532 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.204510    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.215620 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.215592    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbzk\" (UniqueName: \"kubernetes.io/projected/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-api-access-pdbzk\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.417634 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.417605    2583 generic.go:356] "Generic (PLEG): container finished" podID="39644fba-18d3-471e-ac91-efe1b9de34df" containerID="56157cbd0b9dcc451eff1c8c1e584d37c5887ded7773903b50c6c03861c34679" exitCode=2
Sep 17 23:34:15.417770 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.417640    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" event={"ID":"39644fba-18d3-471e-ac91-efe1b9de34df","Type":"ContainerDied","Data":"56157cbd0b9dcc451eff1c8c1e584d37c5887ded7773903b50c6c03861c34679"}
Sep 17 23:34:15.707547 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.707481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835d0960-08b8-47c9-bf25-8b4df6b73001-proxy-tls\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.710057 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.710010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835d0960-08b8-47c9-bf25-8b4df6b73001-proxy-tls\") pod \"success-200-isvc-51963-predictor-55b958c4f8-6sthm\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") " pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:15.911233 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:15.911200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:16.182230 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:16.182204    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"]
Sep 17 23:34:16.183876 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:34:16.183848    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835d0960_08b8_47c9_bf25_8b4df6b73001.slice/crio-fff0559698dc2b4672a58941a13d240f99620176ebcc35b2620d3f4669f7c2ad WatchSource:0}: Error finding container fff0559698dc2b4672a58941a13d240f99620176ebcc35b2620d3f4669f7c2ad: Status 404 returned error can't find the container with id fff0559698dc2b4672a58941a13d240f99620176ebcc35b2620d3f4669f7c2ad
Sep 17 23:34:16.431340 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:16.431301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" event={"ID":"835d0960-08b8-47c9-bf25-8b4df6b73001","Type":"ContainerStarted","Data":"efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2"}
Sep 17 23:34:16.431492 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:16.431395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" event={"ID":"835d0960-08b8-47c9-bf25-8b4df6b73001","Type":"ContainerStarted","Data":"45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf"}
Sep 17 23:34:16.431492 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:16.431435    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:16.431492 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:16.431464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" event={"ID":"835d0960-08b8-47c9-bf25-8b4df6b73001","Type":"ContainerStarted","Data":"fff0559698dc2b4672a58941a13d240f99620176ebcc35b2620d3f4669f7c2ad"}
Sep 17 23:34:16.431671 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:16.431497    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:16.432875 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:16.432834    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:34:16.468501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:16.468363    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podStartSLOduration=2.468347973 podStartE2EDuration="2.468347973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:16.467320039 +0000 UTC m=+1419.401765803" watchObservedRunningTime="2026-09-17 23:34:16.468347973 +0000 UTC m=+1419.402793724"
Sep 17 23:34:17.440402 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.440369    2583 generic.go:356] "Generic (PLEG): container finished" podID="39644fba-18d3-471e-ac91-efe1b9de34df" containerID="eb177c7eaf41c654129e355fa4fc3df0180132985da161fda959849e6dff17f5" exitCode=0
Sep 17 23:34:17.440706 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.440443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" event={"ID":"39644fba-18d3-471e-ac91-efe1b9de34df","Type":"ContainerDied","Data":"eb177c7eaf41c654129e355fa4fc3df0180132985da161fda959849e6dff17f5"}
Sep 17 23:34:17.440866 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.440829    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:34:17.508091 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.508072    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:34:17.623835 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.623773    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39644fba-18d3-471e-ac91-efe1b9de34df-proxy-tls\") pod \"39644fba-18d3-471e-ac91-efe1b9de34df\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") "
Sep 17 23:34:17.623835 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.623819    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pfr2\" (UniqueName: \"kubernetes.io/projected/39644fba-18d3-471e-ac91-efe1b9de34df-kube-api-access-7pfr2\") pod \"39644fba-18d3-471e-ac91-efe1b9de34df\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") "
Sep 17 23:34:17.623973 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.623874    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39644fba-18d3-471e-ac91-efe1b9de34df-kube-rbac-proxy-sar-config\") pod \"39644fba-18d3-471e-ac91-efe1b9de34df\" (UID: \"39644fba-18d3-471e-ac91-efe1b9de34df\") "
Sep 17 23:34:17.624294 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.624272    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39644fba-18d3-471e-ac91-efe1b9de34df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39644fba-18d3-471e-ac91-efe1b9de34df" (UID: "39644fba-18d3-471e-ac91-efe1b9de34df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:34:17.625862 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.625830    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39644fba-18d3-471e-ac91-efe1b9de34df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39644fba-18d3-471e-ac91-efe1b9de34df" (UID: "39644fba-18d3-471e-ac91-efe1b9de34df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:34:17.625971 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.625878    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39644fba-18d3-471e-ac91-efe1b9de34df-kube-api-access-7pfr2" (OuterVolumeSpecName: "kube-api-access-7pfr2") pod "39644fba-18d3-471e-ac91-efe1b9de34df" (UID: "39644fba-18d3-471e-ac91-efe1b9de34df"). InnerVolumeSpecName "kube-api-access-7pfr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:34:17.725333 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.725309    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39644fba-18d3-471e-ac91-efe1b9de34df-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:34:17.725333 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.725331    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfr2\" (UniqueName: \"kubernetes.io/projected/39644fba-18d3-471e-ac91-efe1b9de34df-kube-api-access-7pfr2\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:34:17.725450 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:17.725340    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39644fba-18d3-471e-ac91-efe1b9de34df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:34:18.223888 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:18.223865    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:34:18.391528 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:18.391482    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:34:18.448358 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:18.448331    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"
Sep 17 23:34:18.448790 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:18.448369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r" event={"ID":"39644fba-18d3-471e-ac91-efe1b9de34df","Type":"ContainerDied","Data":"1d79aa5c612f1c1a84ce9f2f9cf6427f0c575c5d7df4336ace33b5ca52e937d1"}
Sep 17 23:34:18.448790 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:18.448436    2583 scope.go:117] "RemoveContainer" containerID="56157cbd0b9dcc451eff1c8c1e584d37c5887ded7773903b50c6c03861c34679"
Sep 17 23:34:18.457472 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:18.457453    2583 scope.go:117] "RemoveContainer" containerID="eb177c7eaf41c654129e355fa4fc3df0180132985da161fda959849e6dff17f5"
Sep 17 23:34:18.470571 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:18.470551    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"]
Sep 17 23:34:18.474798 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:18.474755    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a1f9-predictor-5456b64ccb-vpv2r"]
Sep 17 23:34:19.157052 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:19.157002    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" path="/var/lib/kubelet/pods/39644fba-18d3-471e-ac91-efe1b9de34df/volumes"
Sep 17 23:34:22.445245 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:22.445212    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:34:22.445922 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:22.445886    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:34:23.391684 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:23.391651    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:34:28.392349 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.392315    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:34:28.392772 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.392400    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:28.596553 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.596521    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"]
Sep 17 23:34:28.596951 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.596936    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container"
Sep 17 23:34:28.596995 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.596953    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container"
Sep 17 23:34:28.596995 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.596965    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kube-rbac-proxy"
Sep 17 23:34:28.596995 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.596970    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kube-rbac-proxy"
Sep 17 23:34:28.597107 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.597029    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kube-rbac-proxy"
Sep 17 23:34:28.597107 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.597039    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="39644fba-18d3-471e-ac91-efe1b9de34df" containerName="kserve-container"
Sep 17 23:34:28.601398 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.601378    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:28.604106 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.604083    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d6263-serving-cert\""
Sep 17 23:34:28.604246 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.604227    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d6263-kube-rbac-proxy-sar-config\""
Sep 17 23:34:28.608716 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.608536    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"]
Sep 17 23:34:28.698423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.698353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0752949-cb2d-4408-ab04-40695b6fc449-proxy-tls\") pod \"ensemble-graph-d6263-569646bdf9-hgph9\" (UID: \"a0752949-cb2d-4408-ab04-40695b6fc449\") " pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:28.698552 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.698447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0752949-cb2d-4408-ab04-40695b6fc449-openshift-service-ca-bundle\") pod \"ensemble-graph-d6263-569646bdf9-hgph9\" (UID: \"a0752949-cb2d-4408-ab04-40695b6fc449\") " pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:28.799354 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.799319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0752949-cb2d-4408-ab04-40695b6fc449-openshift-service-ca-bundle\") pod \"ensemble-graph-d6263-569646bdf9-hgph9\" (UID: \"a0752949-cb2d-4408-ab04-40695b6fc449\") " pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:28.799488 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.799362    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0752949-cb2d-4408-ab04-40695b6fc449-proxy-tls\") pod \"ensemble-graph-d6263-569646bdf9-hgph9\" (UID: \"a0752949-cb2d-4408-ab04-40695b6fc449\") " pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:28.799963 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.799942    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0752949-cb2d-4408-ab04-40695b6fc449-openshift-service-ca-bundle\") pod \"ensemble-graph-d6263-569646bdf9-hgph9\" (UID: \"a0752949-cb2d-4408-ab04-40695b6fc449\") " pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:28.801737 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.801714    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0752949-cb2d-4408-ab04-40695b6fc449-proxy-tls\") pod \"ensemble-graph-d6263-569646bdf9-hgph9\" (UID: \"a0752949-cb2d-4408-ab04-40695b6fc449\") " pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:28.912695 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:28.912669    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:29.184307 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:29.184277    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"]
Sep 17 23:34:29.187708 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:34:29.187672    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0752949_cb2d_4408_ab04_40695b6fc449.slice/crio-4a797d24d9a6d9d39d6758b6d024877ba8450fb90be31e1f981ace65b8a38102 WatchSource:0}: Error finding container 4a797d24d9a6d9d39d6758b6d024877ba8450fb90be31e1f981ace65b8a38102: Status 404 returned error can't find the container with id 4a797d24d9a6d9d39d6758b6d024877ba8450fb90be31e1f981ace65b8a38102
Sep 17 23:34:29.501908 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:29.501826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" event={"ID":"a0752949-cb2d-4408-ab04-40695b6fc449","Type":"ContainerStarted","Data":"126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b"}
Sep 17 23:34:29.501908 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:29.501877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" event={"ID":"a0752949-cb2d-4408-ab04-40695b6fc449","Type":"ContainerStarted","Data":"4a797d24d9a6d9d39d6758b6d024877ba8450fb90be31e1f981ace65b8a38102"}
Sep 17 23:34:29.501908 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:29.501904    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:29.520216 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:29.520171    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" podStartSLOduration=1.520157092 podStartE2EDuration="1.520157092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:29.518212111 +0000 UTC m=+1432.452657860" watchObservedRunningTime="2026-09-17 23:34:29.520157092 +0000 UTC m=+1432.454602841"
Sep 17 23:34:32.446044 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:32.445983    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:34:33.391397 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:33.391351    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:34:35.512036 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:35.511990    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:34:38.391518 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:38.391478    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:34:42.446146 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:42.446102    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:34:43.391636 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:43.391597    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:34:44.899410 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:44.899386    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:45.015619 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.015535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-openshift-service-ca-bundle\") pod \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") "
Sep 17 23:34:45.015619 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.015576    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls\") pod \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\" (UID: \"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8\") "
Sep 17 23:34:45.015953 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.015925    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" (UID: "2fb527d6-dff9-4e49-b3ff-242bd1db8fb8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:34:45.017637 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.017617    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" (UID: "2fb527d6-dff9-4e49-b3ff-242bd1db8fb8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:34:45.117166 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.117136    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:34:45.117166 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.117162    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:34:45.571389 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.571355    2583 generic.go:356] "Generic (PLEG): container finished" podID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerID="10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d" exitCode=0
Sep 17 23:34:45.571505 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.571397    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"
Sep 17 23:34:45.571505 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.571441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" event={"ID":"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8","Type":"ContainerDied","Data":"10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d"}
Sep 17 23:34:45.571505 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.571498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95" event={"ID":"2fb527d6-dff9-4e49-b3ff-242bd1db8fb8","Type":"ContainerDied","Data":"8e2bf1d4698b338d3de09b457e51a6e594c7ddc0704a90f5c47a75c853a63936"}
Sep 17 23:34:45.571606 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.571545    2583 scope.go:117] "RemoveContainer" containerID="10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d"
Sep 17 23:34:45.579845 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.579824    2583 scope.go:117] "RemoveContainer" containerID="10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d"
Sep 17 23:34:45.580132 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:34:45.580115    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d\": container with ID starting with 10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d not found: ID does not exist" containerID="10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d"
Sep 17 23:34:45.580186 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.580139    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d"} err="failed to get container status \"10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d\": rpc error: code = NotFound desc = could not find container \"10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d\": container with ID starting with 10541568de149b1914b2ea96bcdf0f5b77b005adf88f4349b95c6a2544f5ce1d not found: ID does not exist"
Sep 17 23:34:45.590328 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.590308    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"]
Sep 17 23:34:45.593953 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:45.593935    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4a1f9-5868fb9865-dwz95"]
Sep 17 23:34:47.157332 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:47.157286    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" path="/var/lib/kubelet/pods/2fb527d6-dff9-4e49-b3ff-242bd1db8fb8/volumes"
Sep 17 23:34:52.446096 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:34:52.446055    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:35:02.446657 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:02.446625    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:35:15.025702 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.025667    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"]
Sep 17 23:35:15.026076 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.026002    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9"
Sep 17 23:35:15.026076 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.026013    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9"
Sep 17 23:35:15.026163 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.026088    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb527d6-dff9-4e49-b3ff-242bd1db8fb8" containerName="sequence-graph-4a1f9"
Sep 17 23:35:15.029109 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.029092    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:15.033196 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.033177    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-51963-kube-rbac-proxy-sar-config\""
Sep 17 23:35:15.033351 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.033336    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-51963-serving-cert\""
Sep 17 23:35:15.043777 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.042914    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"]
Sep 17 23:35:15.122486 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.122455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/591bb5b4-95ed-46d5-aaa3-1489097a0e90-proxy-tls\") pod \"sequence-graph-51963-6984fdf9fc-8m9mp\" (UID: \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\") " pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:15.122633 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.122517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591bb5b4-95ed-46d5-aaa3-1489097a0e90-openshift-service-ca-bundle\") pod \"sequence-graph-51963-6984fdf9fc-8m9mp\" (UID: \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\") " pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:15.223532 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.223502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591bb5b4-95ed-46d5-aaa3-1489097a0e90-openshift-service-ca-bundle\") pod \"sequence-graph-51963-6984fdf9fc-8m9mp\" (UID: \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\") " pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:15.223757 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.223728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/591bb5b4-95ed-46d5-aaa3-1489097a0e90-proxy-tls\") pod \"sequence-graph-51963-6984fdf9fc-8m9mp\" (UID: \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\") " pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:15.223884 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:35:15.223864    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-51963-serving-cert: secret "sequence-graph-51963-serving-cert" not found
Sep 17 23:35:15.223951 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:35:15.223941    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/591bb5b4-95ed-46d5-aaa3-1489097a0e90-proxy-tls podName:591bb5b4-95ed-46d5-aaa3-1489097a0e90 nodeName:}" failed. No retries permitted until 2026-09-17 23:35:15.723925907 +0000 UTC m=+1478.658371635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/591bb5b4-95ed-46d5-aaa3-1489097a0e90-proxy-tls") pod "sequence-graph-51963-6984fdf9fc-8m9mp" (UID: "591bb5b4-95ed-46d5-aaa3-1489097a0e90") : secret "sequence-graph-51963-serving-cert" not found
Sep 17 23:35:15.224157 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.224139    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591bb5b4-95ed-46d5-aaa3-1489097a0e90-openshift-service-ca-bundle\") pod \"sequence-graph-51963-6984fdf9fc-8m9mp\" (UID: \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\") " pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:15.727362 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.727330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/591bb5b4-95ed-46d5-aaa3-1489097a0e90-proxy-tls\") pod \"sequence-graph-51963-6984fdf9fc-8m9mp\" (UID: \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\") " pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:15.730120 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.730093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/591bb5b4-95ed-46d5-aaa3-1489097a0e90-proxy-tls\") pod \"sequence-graph-51963-6984fdf9fc-8m9mp\" (UID: \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\") " pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:15.941855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:15.941831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:16.217403 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:16.217378    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"]
Sep 17 23:35:16.219469 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:35:16.219440    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591bb5b4_95ed_46d5_aaa3_1489097a0e90.slice/crio-6964d5b0669f62c5ac90bab64c7b396c705e91074c774ea4258e01c14b7e8af9 WatchSource:0}: Error finding container 6964d5b0669f62c5ac90bab64c7b396c705e91074c774ea4258e01c14b7e8af9: Status 404 returned error can't find the container with id 6964d5b0669f62c5ac90bab64c7b396c705e91074c774ea4258e01c14b7e8af9
Sep 17 23:35:16.718803 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:16.718763    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" event={"ID":"591bb5b4-95ed-46d5-aaa3-1489097a0e90","Type":"ContainerStarted","Data":"d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611"}
Sep 17 23:35:16.718973 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:16.718844    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:16.718973 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:16.718875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" event={"ID":"591bb5b4-95ed-46d5-aaa3-1489097a0e90","Type":"ContainerStarted","Data":"6964d5b0669f62c5ac90bab64c7b396c705e91074c774ea4258e01c14b7e8af9"}
Sep 17 23:35:16.737938 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:16.737890    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" podStartSLOduration=1.737874851 podStartE2EDuration="1.737874851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:35:16.736367126 +0000 UTC m=+1479.670812876" watchObservedRunningTime="2026-09-17 23:35:16.737874851 +0000 UTC m=+1479.672320601"
Sep 17 23:35:22.731093 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:22.731057    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:35:39.174819 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:39.174788    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:35:39.175349 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:39.175036    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:35:39.178155 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:39.178134    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:35:39.178362 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:35:39.178347    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:40:39.201759 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:40:39.201733    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:40:39.204494 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:40:39.204470    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:40:39.204618 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:40:39.204603    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:40:39.207108 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:40:39.207091    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:42:43.322533 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.322492    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"]
Sep 17 23:42:43.323178 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.322755    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263" containerID="cri-o://126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b" gracePeriod=30
Sep 17 23:42:43.540471 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.540434    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"]
Sep 17 23:42:43.540712 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.540686    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container" containerID="cri-o://77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7" gracePeriod=30
Sep 17 23:42:43.540794 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.540736    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kube-rbac-proxy" containerID="cri-o://6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590" gracePeriod=30
Sep 17 23:42:43.603281 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.603193    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"]
Sep 17 23:42:43.607272 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.607242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.610218 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.610186    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fc189-predictor-serving-cert\""
Sep 17 23:42:43.610550 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.610522    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fc189-kube-rbac-proxy-sar-config\""
Sep 17 23:42:43.622066 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.622003    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"]
Sep 17 23:42:43.666202 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.666157    2583 generic.go:356] "Generic (PLEG): container finished" podID="31174a96-f041-4e06-878b-0e4487e22ca3" containerID="6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590" exitCode=2
Sep 17 23:42:43.666444 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.666216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" event={"ID":"31174a96-f041-4e06-878b-0e4487e22ca3","Type":"ContainerDied","Data":"6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590"}
Sep 17 23:42:43.748988 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.748935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-api-access-jkpbj\") pod \"success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") " pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.749212 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.749094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") " pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.749212 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.749131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4f4541d-671a-4d17-87dc-a4a7471d5268-proxy-tls\") pod \"success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") " pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.851165 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.850512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") " pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.851165 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.850593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4f4541d-671a-4d17-87dc-a4a7471d5268-proxy-tls\") pod \"success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") " pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.851165 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.850764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-api-access-jkpbj\") pod \"success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") " pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.852787 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.851588    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") " pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.857330 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.854388    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4f4541d-671a-4d17-87dc-a4a7471d5268-proxy-tls\") pod \"success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") " pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.861878 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.861848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-api-access-jkpbj\") pod \"success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") " pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:43.920594 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:43.920520    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:44.198767 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:44.198674    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"]
Sep 17 23:42:44.202856 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:42:44.202822    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f4541d_671a_4d17_87dc_a4a7471d5268.slice/crio-b1c8bd8e797652fcd7ee1e9bca1f7bf5fa3860a6beef524e66d882f63a2ab3d7 WatchSource:0}: Error finding container b1c8bd8e797652fcd7ee1e9bca1f7bf5fa3860a6beef524e66d882f63a2ab3d7: Status 404 returned error can't find the container with id b1c8bd8e797652fcd7ee1e9bca1f7bf5fa3860a6beef524e66d882f63a2ab3d7
Sep 17 23:42:44.674079 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:44.674013    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" event={"ID":"f4f4541d-671a-4d17-87dc-a4a7471d5268","Type":"ContainerStarted","Data":"31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf"}
Sep 17 23:42:44.674567 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:44.674120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" event={"ID":"f4f4541d-671a-4d17-87dc-a4a7471d5268","Type":"ContainerStarted","Data":"dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc"}
Sep 17 23:42:44.674567 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:44.674157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:44.674567 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:44.674185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" event={"ID":"f4f4541d-671a-4d17-87dc-a4a7471d5268","Type":"ContainerStarted","Data":"b1c8bd8e797652fcd7ee1e9bca1f7bf5fa3860a6beef524e66d882f63a2ab3d7"}
Sep 17 23:42:44.693973 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:44.693885    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podStartSLOduration=1.6938644269999998 podStartE2EDuration="1.693864427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:42:44.69309961 +0000 UTC m=+1927.627545364" watchObservedRunningTime="2026-09-17 23:42:44.693864427 +0000 UTC m=+1927.628310218"
Sep 17 23:42:45.511486 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:45.511444    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:42:45.678527 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:45.678486    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:45.679944 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:45.679911    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:42:46.683247 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:46.683205    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:42:47.201602 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.201572    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:42:47.274935 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.274838    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls\") pod \"31174a96-f041-4e06-878b-0e4487e22ca3\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") "
Sep 17 23:42:47.274935 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.274923    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td4l7\" (UniqueName: \"kubernetes.io/projected/31174a96-f041-4e06-878b-0e4487e22ca3-kube-api-access-td4l7\") pod \"31174a96-f041-4e06-878b-0e4487e22ca3\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") "
Sep 17 23:42:47.275172 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.274960    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31174a96-f041-4e06-878b-0e4487e22ca3-kube-rbac-proxy-sar-config\") pod \"31174a96-f041-4e06-878b-0e4487e22ca3\" (UID: \"31174a96-f041-4e06-878b-0e4487e22ca3\") "
Sep 17 23:42:47.275347 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.275326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31174a96-f041-4e06-878b-0e4487e22ca3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31174a96-f041-4e06-878b-0e4487e22ca3" (UID: "31174a96-f041-4e06-878b-0e4487e22ca3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:42:47.277191 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.277147    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31174a96-f041-4e06-878b-0e4487e22ca3" (UID: "31174a96-f041-4e06-878b-0e4487e22ca3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:42:47.277341 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.277256    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31174a96-f041-4e06-878b-0e4487e22ca3-kube-api-access-td4l7" (OuterVolumeSpecName: "kube-api-access-td4l7") pod "31174a96-f041-4e06-878b-0e4487e22ca3" (UID: "31174a96-f041-4e06-878b-0e4487e22ca3"). InnerVolumeSpecName "kube-api-access-td4l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:42:47.375517 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.375478    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td4l7\" (UniqueName: \"kubernetes.io/projected/31174a96-f041-4e06-878b-0e4487e22ca3-kube-api-access-td4l7\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:42:47.375517 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.375513    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31174a96-f041-4e06-878b-0e4487e22ca3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:42:47.375517 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.375523    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31174a96-f041-4e06-878b-0e4487e22ca3-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:42:47.689970 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.689879    2583 generic.go:356] "Generic (PLEG): container finished" podID="31174a96-f041-4e06-878b-0e4487e22ca3" containerID="77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7" exitCode=0
Sep 17 23:42:47.689970 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.689930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" event={"ID":"31174a96-f041-4e06-878b-0e4487e22ca3","Type":"ContainerDied","Data":"77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7"}
Sep 17 23:42:47.690433 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.689987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj" event={"ID":"31174a96-f041-4e06-878b-0e4487e22ca3","Type":"ContainerDied","Data":"351121b195f29295333cd1edc7f33e7588b9ec2e7f127d3fa9ef2b4131c70b9a"}
Sep 17 23:42:47.690433 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.690013    2583 scope.go:117] "RemoveContainer" containerID="6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590"
Sep 17 23:42:47.690433 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.689935    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"
Sep 17 23:42:47.699219 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.699196    2583 scope.go:117] "RemoveContainer" containerID="77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7"
Sep 17 23:42:47.707814 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.707789    2583 scope.go:117] "RemoveContainer" containerID="6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590"
Sep 17 23:42:47.708229 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:42:47.708206    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590\": container with ID starting with 6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590 not found: ID does not exist" containerID="6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590"
Sep 17 23:42:47.708327 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.708238    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590"} err="failed to get container status \"6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590\": rpc error: code = NotFound desc = could not find container \"6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590\": container with ID starting with 6233f9a11009ea3cc61f1c314d0dc750df857d74784a00809ec6f5f417176590 not found: ID does not exist"
Sep 17 23:42:47.708327 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.708256    2583 scope.go:117] "RemoveContainer" containerID="77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7"
Sep 17 23:42:47.708525 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:42:47.708507    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7\": container with ID starting with 77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7 not found: ID does not exist" containerID="77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7"
Sep 17 23:42:47.708562 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.708531    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7"} err="failed to get container status \"77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7\": rpc error: code = NotFound desc = could not find container \"77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7\": container with ID starting with 77942a4c52227c2a264f111aa3af44252a93536b2fc6c28492556bc656cf99b7 not found: ID does not exist"
Sep 17 23:42:47.714403 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.714365    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"]
Sep 17 23:42:47.721398 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:47.721360    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d6263-predictor-5bd97bfcc6-l5bmj"]
Sep 17 23:42:49.157946 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:49.157902    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" path="/var/lib/kubelet/pods/31174a96-f041-4e06-878b-0e4487e22ca3/volumes"
Sep 17 23:42:50.511144 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:50.511100    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:42:51.687683 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:51.687653    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:42:51.688123 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:51.688092    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:42:55.510899 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:55.510842    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:42:55.511371 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:42:55.510976    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:43:00.511007 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:00.510961    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:43:01.688451 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:01.688404    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:43:05.511111 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:05.511071    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:43:10.510879 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:10.510828    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:43:11.688674 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:11.688629    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:43:13.491212 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.491177    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:43:13.590956 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.590852    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0752949-cb2d-4408-ab04-40695b6fc449-proxy-tls\") pod \"a0752949-cb2d-4408-ab04-40695b6fc449\" (UID: \"a0752949-cb2d-4408-ab04-40695b6fc449\") "
Sep 17 23:43:13.590956 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.590906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0752949-cb2d-4408-ab04-40695b6fc449-openshift-service-ca-bundle\") pod \"a0752949-cb2d-4408-ab04-40695b6fc449\" (UID: \"a0752949-cb2d-4408-ab04-40695b6fc449\") "
Sep 17 23:43:13.591338 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.591311    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0752949-cb2d-4408-ab04-40695b6fc449-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a0752949-cb2d-4408-ab04-40695b6fc449" (UID: "a0752949-cb2d-4408-ab04-40695b6fc449"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:43:13.593299 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.593268    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0752949-cb2d-4408-ab04-40695b6fc449-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0752949-cb2d-4408-ab04-40695b6fc449" (UID: "a0752949-cb2d-4408-ab04-40695b6fc449"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:43:13.692072 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.692008    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0752949-cb2d-4408-ab04-40695b6fc449-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:43:13.692072 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.692067    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0752949-cb2d-4408-ab04-40695b6fc449-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:43:13.812315 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.812274    2583 generic.go:356] "Generic (PLEG): container finished" podID="a0752949-cb2d-4408-ab04-40695b6fc449" containerID="126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b" exitCode=0
Sep 17 23:43:13.812315 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.812325    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"
Sep 17 23:43:13.812561 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.812366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" event={"ID":"a0752949-cb2d-4408-ab04-40695b6fc449","Type":"ContainerDied","Data":"126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b"}
Sep 17 23:43:13.812561 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.812428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9" event={"ID":"a0752949-cb2d-4408-ab04-40695b6fc449","Type":"ContainerDied","Data":"4a797d24d9a6d9d39d6758b6d024877ba8450fb90be31e1f981ace65b8a38102"}
Sep 17 23:43:13.812561 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.812452    2583 scope.go:117] "RemoveContainer" containerID="126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b"
Sep 17 23:43:13.822416 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.822396    2583 scope.go:117] "RemoveContainer" containerID="126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b"
Sep 17 23:43:13.822789 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:43:13.822766    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b\": container with ID starting with 126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b not found: ID does not exist" containerID="126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b"
Sep 17 23:43:13.822854 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.822800    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b"} err="failed to get container status \"126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b\": rpc error: code = NotFound desc = could not find container \"126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b\": container with ID starting with 126724e63dacaa06cc742de0077ed8e8d0de28ade9e8cb63c6222dac5def941b not found: ID does not exist"
Sep 17 23:43:13.835477 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.835434    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"]
Sep 17 23:43:13.839225 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:13.839188    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d6263-569646bdf9-hgph9"]
Sep 17 23:43:15.157229 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:15.157183    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" path="/var/lib/kubelet/pods/a0752949-cb2d-4408-ab04-40695b6fc449/volumes"
Sep 17 23:43:21.688476 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:21.688430    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:43:29.695919 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:29.695873    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"]
Sep 17 23:43:29.696316 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:29.696179    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963" containerID="cri-o://d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611" gracePeriod=30
Sep 17 23:43:29.862959 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:29.862916    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"]
Sep 17 23:43:29.863283 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:29.863252    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container" containerID="cri-o://45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf" gracePeriod=30
Sep 17 23:43:29.863375 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:29.863282    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kube-rbac-proxy" containerID="cri-o://efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2" gracePeriod=30
Sep 17 23:43:30.006890 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.006844    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"]
Sep 17 23:43:30.007304 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.007288    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263"
Sep 17 23:43:30.007355 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.007306    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263"
Sep 17 23:43:30.007355 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.007322    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container"
Sep 17 23:43:30.007355 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.007328    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container"
Sep 17 23:43:30.007355 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.007335    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kube-rbac-proxy"
Sep 17 23:43:30.007355 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.007341    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kube-rbac-proxy"
Sep 17 23:43:30.007521 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.007395    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kube-rbac-proxy"
Sep 17 23:43:30.007521 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.007405    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31174a96-f041-4e06-878b-0e4487e22ca3" containerName="kserve-container"
Sep 17 23:43:30.007521 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.007412    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0752949-cb2d-4408-ab04-40695b6fc449" containerName="ensemble-graph-d6263"
Sep 17 23:43:30.010783 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.010753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.013349 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.013317    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-57566-predictor-serving-cert\""
Sep 17 23:43:30.013499 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.013368    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-57566-kube-rbac-proxy-sar-config\""
Sep 17 23:43:30.021984 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.021946    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"]
Sep 17 23:43:30.135709 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.135660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c151445-b6ed-41c4-9e32-362ee62596ed-proxy-tls\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.135907 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.135717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.135907 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.135795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jdpt\" (UniqueName: \"kubernetes.io/projected/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-api-access-6jdpt\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.236929 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.236828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c151445-b6ed-41c4-9e32-362ee62596ed-proxy-tls\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.236929 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.236891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.236929 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.236920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jdpt\" (UniqueName: \"kubernetes.io/projected/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-api-access-6jdpt\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.237228 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:43:30.237034    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-57566-predictor-serving-cert: secret "success-200-isvc-57566-predictor-serving-cert" not found
Sep 17 23:43:30.237228 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:43:30.237104    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c151445-b6ed-41c4-9e32-362ee62596ed-proxy-tls podName:1c151445-b6ed-41c4-9e32-362ee62596ed nodeName:}" failed. No retries permitted until 2026-09-17 23:43:30.737085239 +0000 UTC m=+1973.671530974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1c151445-b6ed-41c4-9e32-362ee62596ed-proxy-tls") pod "success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" (UID: "1c151445-b6ed-41c4-9e32-362ee62596ed") : secret "success-200-isvc-57566-predictor-serving-cert" not found
Sep 17 23:43:30.237665 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.237641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.248802 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.248768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jdpt\" (UniqueName: \"kubernetes.io/projected/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-api-access-6jdpt\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.743003 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.742950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c151445-b6ed-41c4-9e32-362ee62596ed-proxy-tls\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.745645 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.745605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c151445-b6ed-41c4-9e32-362ee62596ed-proxy-tls\") pod \"success-200-isvc-57566-predictor-6949f9f8f6-9bgg7\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") " pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:30.896926 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.896889    2583 generic.go:356] "Generic (PLEG): container finished" podID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerID="efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2" exitCode=2
Sep 17 23:43:30.897123 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.896934    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" event={"ID":"835d0960-08b8-47c9-bf25-8b4df6b73001","Type":"ContainerDied","Data":"efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2"}
Sep 17 23:43:30.924666 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:30.924578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:31.201679 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:31.201649    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"]
Sep 17 23:43:31.203895 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:43:31.203856    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c151445_b6ed_41c4_9e32_362ee62596ed.slice/crio-9e57f59fc88a0ea1f2e308ea4d7cdd5b043c84a695b360fcc61902cd61c29194 WatchSource:0}: Error finding container 9e57f59fc88a0ea1f2e308ea4d7cdd5b043c84a695b360fcc61902cd61c29194: Status 404 returned error can't find the container with id 9e57f59fc88a0ea1f2e308ea4d7cdd5b043c84a695b360fcc61902cd61c29194
Sep 17 23:43:31.688447 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:31.688404    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 23:43:31.904007 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:31.903965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" event={"ID":"1c151445-b6ed-41c4-9e32-362ee62596ed","Type":"ContainerStarted","Data":"9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f"}
Sep 17 23:43:31.904409 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:31.904053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" event={"ID":"1c151445-b6ed-41c4-9e32-362ee62596ed","Type":"ContainerStarted","Data":"ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8"}
Sep 17 23:43:31.904409 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:31.904086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" event={"ID":"1c151445-b6ed-41c4-9e32-362ee62596ed","Type":"ContainerStarted","Data":"9e57f59fc88a0ea1f2e308ea4d7cdd5b043c84a695b360fcc61902cd61c29194"}
Sep 17 23:43:31.904409 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:31.904230    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:31.939005 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:31.938875    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podStartSLOduration=2.938856374 podStartE2EDuration="2.938856374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:43:31.93832879 +0000 UTC m=+1974.872774539" watchObservedRunningTime="2026-09-17 23:43:31.938856374 +0000 UTC m=+1974.873302124"
Sep 17 23:43:32.441827 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:32.441778    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 23:43:32.446304 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:32.446266    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 23:43:32.729862 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:32.729761    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:43:32.909115 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:32.909079    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:32.910669 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:32.910636    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:43:33.526466 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.526430    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:43:33.668509 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.668402    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835d0960-08b8-47c9-bf25-8b4df6b73001-proxy-tls\") pod \"835d0960-08b8-47c9-bf25-8b4df6b73001\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") "
Sep 17 23:43:33.668509 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.668471    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-rbac-proxy-sar-config\") pod \"835d0960-08b8-47c9-bf25-8b4df6b73001\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") "
Sep 17 23:43:33.668744 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.668521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdbzk\" (UniqueName: \"kubernetes.io/projected/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-api-access-pdbzk\") pod \"835d0960-08b8-47c9-bf25-8b4df6b73001\" (UID: \"835d0960-08b8-47c9-bf25-8b4df6b73001\") "
Sep 17 23:43:33.668932 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.668909    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "835d0960-08b8-47c9-bf25-8b4df6b73001" (UID: "835d0960-08b8-47c9-bf25-8b4df6b73001"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:43:33.670722 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.670683    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835d0960-08b8-47c9-bf25-8b4df6b73001-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "835d0960-08b8-47c9-bf25-8b4df6b73001" (UID: "835d0960-08b8-47c9-bf25-8b4df6b73001"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:43:33.670813 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.670702    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-api-access-pdbzk" (OuterVolumeSpecName: "kube-api-access-pdbzk") pod "835d0960-08b8-47c9-bf25-8b4df6b73001" (UID: "835d0960-08b8-47c9-bf25-8b4df6b73001"). InnerVolumeSpecName "kube-api-access-pdbzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:43:33.769541 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.769492    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdbzk\" (UniqueName: \"kubernetes.io/projected/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-api-access-pdbzk\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:43:33.769541 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.769535    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/835d0960-08b8-47c9-bf25-8b4df6b73001-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:43:33.769541 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.769551    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/835d0960-08b8-47c9-bf25-8b4df6b73001-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:43:33.915497 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.915456    2583 generic.go:356] "Generic (PLEG): container finished" podID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerID="45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf" exitCode=0
Sep 17 23:43:33.915876 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.915509    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"
Sep 17 23:43:33.915876 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.915539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" event={"ID":"835d0960-08b8-47c9-bf25-8b4df6b73001","Type":"ContainerDied","Data":"45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf"}
Sep 17 23:43:33.915876 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.915607    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm" event={"ID":"835d0960-08b8-47c9-bf25-8b4df6b73001","Type":"ContainerDied","Data":"fff0559698dc2b4672a58941a13d240f99620176ebcc35b2620d3f4669f7c2ad"}
Sep 17 23:43:33.915876 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.915646    2583 scope.go:117] "RemoveContainer" containerID="efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2"
Sep 17 23:43:33.916184 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.916154    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:43:33.925333 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.925302    2583 scope.go:117] "RemoveContainer" containerID="45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf"
Sep 17 23:43:33.934720 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.934695    2583 scope.go:117] "RemoveContainer" containerID="efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2"
Sep 17 23:43:33.935100 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:43:33.935077    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2\": container with ID starting with efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2 not found: ID does not exist" containerID="efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2"
Sep 17 23:43:33.935188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.935111    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2"} err="failed to get container status \"efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2\": rpc error: code = NotFound desc = could not find container \"efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2\": container with ID starting with efeb67ea9b0f03ee280e6dbe7a6c8d685ae427b26fc25ca23cdfe1e89c4c23f2 not found: ID does not exist"
Sep 17 23:43:33.935188 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.935131    2583 scope.go:117] "RemoveContainer" containerID="45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf"
Sep 17 23:43:33.935408 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:43:33.935383    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf\": container with ID starting with 45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf not found: ID does not exist" containerID="45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf"
Sep 17 23:43:33.935449 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.935414    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf"} err="failed to get container status \"45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf\": rpc error: code = NotFound desc = could not find container \"45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf\": container with ID starting with 45fc5ad1a9d9dac4671696083a604972399fe9b22cb58d34ef87d0eed2c3b8cf not found: ID does not exist"
Sep 17 23:43:33.941437 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.941402    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"]
Sep 17 23:43:33.945946 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:33.945905    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-51963-predictor-55b958c4f8-6sthm"]
Sep 17 23:43:35.157570 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:35.157520    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" path="/var/lib/kubelet/pods/835d0960-08b8-47c9-bf25-8b4df6b73001/volumes"
Sep 17 23:43:37.730011 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:37.729968    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:43:38.925484 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:38.925450    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:43:38.925986 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:38.925958    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:43:41.689008 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:41.688973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:43:42.729657 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:42.729610    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:43:42.730182 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:42.729720    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:43:47.730161 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:47.730115    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:43:48.926731 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:48.926681    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:43:52.730373 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:52.730326    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:43:53.579847 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.579753    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"]
Sep 17 23:43:53.580164 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.580148    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kube-rbac-proxy"
Sep 17 23:43:53.580215 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.580165    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kube-rbac-proxy"
Sep 17 23:43:53.580215 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.580178    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container"
Sep 17 23:43:53.580215 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.580184    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container"
Sep 17 23:43:53.580313 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.580239    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kserve-container"
Sep 17 23:43:53.580313 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.580248    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="835d0960-08b8-47c9-bf25-8b4df6b73001" containerName="kube-rbac-proxy"
Sep 17 23:43:53.584814 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.584785    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:53.587711 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.587679    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-fc189-kube-rbac-proxy-sar-config\""
Sep 17 23:43:53.587855 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.587729    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-fc189-serving-cert\""
Sep 17 23:43:53.596427 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.596392    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"]
Sep 17 23:43:53.627093 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.627042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450701f9-bf58-4704-8dbc-2cf4c3274efa-openshift-service-ca-bundle\") pod \"splitter-graph-fc189-8457d9b884-rsm8p\" (UID: \"450701f9-bf58-4704-8dbc-2cf4c3274efa\") " pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:53.627093 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.627104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/450701f9-bf58-4704-8dbc-2cf4c3274efa-proxy-tls\") pod \"splitter-graph-fc189-8457d9b884-rsm8p\" (UID: \"450701f9-bf58-4704-8dbc-2cf4c3274efa\") " pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:53.727927 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.727871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/450701f9-bf58-4704-8dbc-2cf4c3274efa-proxy-tls\") pod \"splitter-graph-fc189-8457d9b884-rsm8p\" (UID: \"450701f9-bf58-4704-8dbc-2cf4c3274efa\") " pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:53.728151 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.727969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450701f9-bf58-4704-8dbc-2cf4c3274efa-openshift-service-ca-bundle\") pod \"splitter-graph-fc189-8457d9b884-rsm8p\" (UID: \"450701f9-bf58-4704-8dbc-2cf4c3274efa\") " pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:53.728151 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:43:53.728069    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-fc189-serving-cert: secret "splitter-graph-fc189-serving-cert" not found
Sep 17 23:43:53.728242 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:43:53.728167    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/450701f9-bf58-4704-8dbc-2cf4c3274efa-proxy-tls podName:450701f9-bf58-4704-8dbc-2cf4c3274efa nodeName:}" failed. No retries permitted until 2026-09-17 23:43:54.228144028 +0000 UTC m=+1997.162589779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/450701f9-bf58-4704-8dbc-2cf4c3274efa-proxy-tls") pod "splitter-graph-fc189-8457d9b884-rsm8p" (UID: "450701f9-bf58-4704-8dbc-2cf4c3274efa") : secret "splitter-graph-fc189-serving-cert" not found
Sep 17 23:43:53.728725 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:53.728703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450701f9-bf58-4704-8dbc-2cf4c3274efa-openshift-service-ca-bundle\") pod \"splitter-graph-fc189-8457d9b884-rsm8p\" (UID: \"450701f9-bf58-4704-8dbc-2cf4c3274efa\") " pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:54.232673 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:54.232631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/450701f9-bf58-4704-8dbc-2cf4c3274efa-proxy-tls\") pod \"splitter-graph-fc189-8457d9b884-rsm8p\" (UID: \"450701f9-bf58-4704-8dbc-2cf4c3274efa\") " pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:54.235287 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:54.235260    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/450701f9-bf58-4704-8dbc-2cf4c3274efa-proxy-tls\") pod \"splitter-graph-fc189-8457d9b884-rsm8p\" (UID: \"450701f9-bf58-4704-8dbc-2cf4c3274efa\") " pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:54.496101 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:54.495994    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:54.781903 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:54.781867    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"]
Sep 17 23:43:54.783968 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:43:54.783933    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450701f9_bf58_4704_8dbc_2cf4c3274efa.slice/crio-b6b9ab55c702f345643c2d9504847fbdf7e2533d754ea80adf6b5f1f1d06dabd WatchSource:0}: Error finding container b6b9ab55c702f345643c2d9504847fbdf7e2533d754ea80adf6b5f1f1d06dabd: Status 404 returned error can't find the container with id b6b9ab55c702f345643c2d9504847fbdf7e2533d754ea80adf6b5f1f1d06dabd
Sep 17 23:43:55.021691 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:55.021645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" event={"ID":"450701f9-bf58-4704-8dbc-2cf4c3274efa","Type":"ContainerStarted","Data":"71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1"}
Sep 17 23:43:55.021931 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:55.021709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" event={"ID":"450701f9-bf58-4704-8dbc-2cf4c3274efa","Type":"ContainerStarted","Data":"b6b9ab55c702f345643c2d9504847fbdf7e2533d754ea80adf6b5f1f1d06dabd"}
Sep 17 23:43:55.021931 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:55.021796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:43:55.044696 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:55.044582    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" podStartSLOduration=2.044558806 podStartE2EDuration="2.044558806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:43:55.0443107 +0000 UTC m=+1997.978756450" watchObservedRunningTime="2026-09-17 23:43:55.044558806 +0000 UTC m=+1997.979004833"
Sep 17 23:43:57.729978 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:57.729934    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:43:58.926877 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:58.926835    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:43:59.895749 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:59.895691    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:43:59.978295 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:59.978179    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/591bb5b4-95ed-46d5-aaa3-1489097a0e90-proxy-tls\") pod \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\" (UID: \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\") "
Sep 17 23:43:59.978720 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:59.978298    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591bb5b4-95ed-46d5-aaa3-1489097a0e90-openshift-service-ca-bundle\") pod \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\" (UID: \"591bb5b4-95ed-46d5-aaa3-1489097a0e90\") "
Sep 17 23:43:59.978720 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:59.978676    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591bb5b4-95ed-46d5-aaa3-1489097a0e90-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "591bb5b4-95ed-46d5-aaa3-1489097a0e90" (UID: "591bb5b4-95ed-46d5-aaa3-1489097a0e90"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:43:59.980498 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:43:59.980471    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591bb5b4-95ed-46d5-aaa3-1489097a0e90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "591bb5b4-95ed-46d5-aaa3-1489097a0e90" (UID: "591bb5b4-95ed-46d5-aaa3-1489097a0e90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:00.046036 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.045981    2583 generic.go:356] "Generic (PLEG): container finished" podID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerID="d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611" exitCode=0
Sep 17 23:44:00.046210 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.046045    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"
Sep 17 23:44:00.046210 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.046063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" event={"ID":"591bb5b4-95ed-46d5-aaa3-1489097a0e90","Type":"ContainerDied","Data":"d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611"}
Sep 17 23:44:00.046210 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.046140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp" event={"ID":"591bb5b4-95ed-46d5-aaa3-1489097a0e90","Type":"ContainerDied","Data":"6964d5b0669f62c5ac90bab64c7b396c705e91074c774ea4258e01c14b7e8af9"}
Sep 17 23:44:00.046210 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.046163    2583 scope.go:117] "RemoveContainer" containerID="d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611"
Sep 17 23:44:00.055758 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.055736    2583 scope.go:117] "RemoveContainer" containerID="d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611"
Sep 17 23:44:00.056142 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:44:00.056116    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611\": container with ID starting with d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611 not found: ID does not exist" containerID="d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611"
Sep 17 23:44:00.056222 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.056152    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611"} err="failed to get container status \"d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611\": rpc error: code = NotFound desc = could not find container \"d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611\": container with ID starting with d0b49f4bcf2b49cc9750afbf21b315bdccea5f7af9c463ca8033d848583ed611 not found: ID does not exist"
Sep 17 23:44:00.070454 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.070413    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"]
Sep 17 23:44:00.075767 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.075731    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-51963-6984fdf9fc-8m9mp"]
Sep 17 23:44:00.079241 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.079213    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/591bb5b4-95ed-46d5-aaa3-1489097a0e90-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:44:00.079344 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:00.079268    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/591bb5b4-95ed-46d5-aaa3-1489097a0e90-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:44:01.032529 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:01.032496    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:44:01.158087 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:01.158050    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" path="/var/lib/kubelet/pods/591bb5b4-95ed-46d5-aaa3-1489097a0e90/volumes"
Sep 17 23:44:03.646696 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.646654    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"]
Sep 17 23:44:03.647225 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.646858    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189" containerID="cri-o://71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1" gracePeriod=30
Sep 17 23:44:03.759930 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.759888    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"]
Sep 17 23:44:03.760211 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.760178    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container" containerID="cri-o://dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc" gracePeriod=30
Sep 17 23:44:03.760484 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.760237    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kube-rbac-proxy" containerID="cri-o://31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf" gracePeriod=30
Sep 17 23:44:03.908780 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.908683    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"]
Sep 17 23:44:03.909101 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.909085    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963"
Sep 17 23:44:03.909154 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.909104    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963"
Sep 17 23:44:03.909190 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.909154    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="591bb5b4-95ed-46d5-aaa3-1489097a0e90" containerName="sequence-graph-51963"
Sep 17 23:44:03.912316 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.912293    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:03.916773 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.916732    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-daa97-predictor-serving-cert\""
Sep 17 23:44:03.916773 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.916747    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-daa97-kube-rbac-proxy-sar-config\""
Sep 17 23:44:03.929860 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:03.929823    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"]
Sep 17 23:44:04.012628 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.012579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.012628 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.012630    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb2de6d8-6144-4506-aac2-678e98ea3f36-proxy-tls\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.012935 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.012663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnsnq\" (UniqueName: \"kubernetes.io/projected/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-api-access-mnsnq\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.069133 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.069092    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerID="31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf" exitCode=2
Sep 17 23:44:04.069318 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.069164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" event={"ID":"f4f4541d-671a-4d17-87dc-a4a7471d5268","Type":"ContainerDied","Data":"31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf"}
Sep 17 23:44:04.113715 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.113669    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb2de6d8-6144-4506-aac2-678e98ea3f36-proxy-tls\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.113923 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.113730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnsnq\" (UniqueName: \"kubernetes.io/projected/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-api-access-mnsnq\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.113923 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.113826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.113923 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:44:04.113849    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-daa97-predictor-serving-cert: secret "success-200-isvc-daa97-predictor-serving-cert" not found
Sep 17 23:44:04.113923 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:44:04.113920    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb2de6d8-6144-4506-aac2-678e98ea3f36-proxy-tls podName:eb2de6d8-6144-4506-aac2-678e98ea3f36 nodeName:}" failed. No retries permitted until 2026-09-17 23:44:04.6138991 +0000 UTC m=+2007.548344836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eb2de6d8-6144-4506-aac2-678e98ea3f36-proxy-tls") pod "success-200-isvc-daa97-predictor-845479f546-9wxxx" (UID: "eb2de6d8-6144-4506-aac2-678e98ea3f36") : secret "success-200-isvc-daa97-predictor-serving-cert" not found
Sep 17 23:44:04.114466 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.114444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.125140 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.125104    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnsnq\" (UniqueName: \"kubernetes.io/projected/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-api-access-mnsnq\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.619097 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.619050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb2de6d8-6144-4506-aac2-678e98ea3f36-proxy-tls\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.621718 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.621687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb2de6d8-6144-4506-aac2-678e98ea3f36-proxy-tls\") pod \"success-200-isvc-daa97-predictor-845479f546-9wxxx\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") " pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:04.825869 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:04.825828    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:05.116373 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:05.116341    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"]
Sep 17 23:44:05.119155 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:44:05.119119    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2de6d8_6144_4506_aac2_678e98ea3f36.slice/crio-ea07f49533cd0b3d041bb7a842a285ba3f8c14176f4afdffefc99e224e20050c WatchSource:0}: Error finding container ea07f49533cd0b3d041bb7a842a285ba3f8c14176f4afdffefc99e224e20050c: Status 404 returned error can't find the container with id ea07f49533cd0b3d041bb7a842a285ba3f8c14176f4afdffefc99e224e20050c
Sep 17 23:44:06.030501 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:06.030455    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:44:06.082139 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:06.082095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" event={"ID":"eb2de6d8-6144-4506-aac2-678e98ea3f36","Type":"ContainerStarted","Data":"a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2"}
Sep 17 23:44:06.082345 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:06.082177    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" event={"ID":"eb2de6d8-6144-4506-aac2-678e98ea3f36","Type":"ContainerStarted","Data":"301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d"}
Sep 17 23:44:06.082400 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:06.082348    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:06.082435 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:06.082426    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:06.082490 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:06.082465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" event={"ID":"eb2de6d8-6144-4506-aac2-678e98ea3f36","Type":"ContainerStarted","Data":"ea07f49533cd0b3d041bb7a842a285ba3f8c14176f4afdffefc99e224e20050c"}
Sep 17 23:44:06.083934 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:06.083901    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 23:44:06.102654 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:06.102594    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podStartSLOduration=3.102576747 podStartE2EDuration="3.102576747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:06.100719284 +0000 UTC m=+2009.035165057" watchObservedRunningTime="2026-09-17 23:44:06.102576747 +0000 UTC m=+2009.037022497"
Sep 17 23:44:06.683843 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:06.683796    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 17 23:44:07.087947 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.087898    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 23:44:07.521440 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.521397    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:44:07.650888 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.650790    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-api-access-jkpbj\") pod \"f4f4541d-671a-4d17-87dc-a4a7471d5268\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") "
Sep 17 23:44:07.650888 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.650845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4f4541d-671a-4d17-87dc-a4a7471d5268-proxy-tls\") pod \"f4f4541d-671a-4d17-87dc-a4a7471d5268\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") "
Sep 17 23:44:07.651136 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.650906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-rbac-proxy-sar-config\") pod \"f4f4541d-671a-4d17-87dc-a4a7471d5268\" (UID: \"f4f4541d-671a-4d17-87dc-a4a7471d5268\") "
Sep 17 23:44:07.651366 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.651333    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4f4541d-671a-4d17-87dc-a4a7471d5268" (UID: "f4f4541d-671a-4d17-87dc-a4a7471d5268"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:44:07.653070 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.653037    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f4541d-671a-4d17-87dc-a4a7471d5268-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4f4541d-671a-4d17-87dc-a4a7471d5268" (UID: "f4f4541d-671a-4d17-87dc-a4a7471d5268"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:07.653192 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.653117    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-api-access-jkpbj" (OuterVolumeSpecName: "kube-api-access-jkpbj") pod "f4f4541d-671a-4d17-87dc-a4a7471d5268" (UID: "f4f4541d-671a-4d17-87dc-a4a7471d5268"). InnerVolumeSpecName "kube-api-access-jkpbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:44:07.751931 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.751888    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkpbj\" (UniqueName: \"kubernetes.io/projected/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-api-access-jkpbj\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:44:07.751931 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.751922    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4f4541d-671a-4d17-87dc-a4a7471d5268-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:44:07.751931 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:07.751933    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4f4541d-671a-4d17-87dc-a4a7471d5268-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:44:08.094184 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.094140    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerID="dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc" exitCode=0
Sep 17 23:44:08.094614 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.094203    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"
Sep 17 23:44:08.094614 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.094226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" event={"ID":"f4f4541d-671a-4d17-87dc-a4a7471d5268","Type":"ContainerDied","Data":"dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc"}
Sep 17 23:44:08.094614 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.094299    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn" event={"ID":"f4f4541d-671a-4d17-87dc-a4a7471d5268","Type":"ContainerDied","Data":"b1c8bd8e797652fcd7ee1e9bca1f7bf5fa3860a6beef524e66d882f63a2ab3d7"}
Sep 17 23:44:08.094614 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.094330    2583 scope.go:117] "RemoveContainer" containerID="31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf"
Sep 17 23:44:08.103699 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.103680    2583 scope.go:117] "RemoveContainer" containerID="dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc"
Sep 17 23:44:08.112460 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.112431    2583 scope.go:117] "RemoveContainer" containerID="31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf"
Sep 17 23:44:08.112828 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:44:08.112807    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf\": container with ID starting with 31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf not found: ID does not exist" containerID="31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf"
Sep 17 23:44:08.112887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.112849    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf"} err="failed to get container status \"31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf\": rpc error: code = NotFound desc = could not find container \"31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf\": container with ID starting with 31187fe9b6467b04d042743ea0dcd6cee8f849b9d6d0da3f26d4f2518cccd3cf not found: ID does not exist"
Sep 17 23:44:08.112887 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.112872    2583 scope.go:117] "RemoveContainer" containerID="dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc"
Sep 17 23:44:08.113228 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:44:08.113200    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc\": container with ID starting with dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc not found: ID does not exist" containerID="dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc"
Sep 17 23:44:08.113350 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.113233    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc"} err="failed to get container status \"dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc\": rpc error: code = NotFound desc = could not find container \"dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc\": container with ID starting with dc103ce006548d70cb55d72a80e4d3b0992770e846b11dedbe036014d18d67fc not found: ID does not exist"
Sep 17 23:44:08.118644 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.118602    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"]
Sep 17 23:44:08.121728 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.121687    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fc189-predictor-8b7b5ddbd-lscbn"]
Sep 17 23:44:08.926679 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:08.926637    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:44:09.157914 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:09.157876    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" path="/var/lib/kubelet/pods/f4f4541d-671a-4d17-87dc-a4a7471d5268/volumes"
Sep 17 23:44:11.030882 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:11.030838    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:44:12.092668 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:12.092634    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:44:12.093184 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:12.093154    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 23:44:16.030255 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:16.030212    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:44:16.030713 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:16.030335    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:44:18.926376 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:18.926323    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 23:44:21.030756 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:21.030708    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:44:22.093322 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:22.093279    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 23:44:26.030549 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:26.030504    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:44:28.927196 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:28.927158    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 17 23:44:31.030581 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:31.030543    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:44:32.093196 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:32.093155    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 23:44:33.829536 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:33.829504    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:44:33.977356 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:33.977266    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/450701f9-bf58-4704-8dbc-2cf4c3274efa-proxy-tls\") pod \"450701f9-bf58-4704-8dbc-2cf4c3274efa\" (UID: \"450701f9-bf58-4704-8dbc-2cf4c3274efa\") "
Sep 17 23:44:33.977356 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:33.977315    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450701f9-bf58-4704-8dbc-2cf4c3274efa-openshift-service-ca-bundle\") pod \"450701f9-bf58-4704-8dbc-2cf4c3274efa\" (UID: \"450701f9-bf58-4704-8dbc-2cf4c3274efa\") "
Sep 17 23:44:33.977741 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:33.977716    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450701f9-bf58-4704-8dbc-2cf4c3274efa-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "450701f9-bf58-4704-8dbc-2cf4c3274efa" (UID: "450701f9-bf58-4704-8dbc-2cf4c3274efa"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:44:33.979539 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:33.979505    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/450701f9-bf58-4704-8dbc-2cf4c3274efa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "450701f9-bf58-4704-8dbc-2cf4c3274efa" (UID: "450701f9-bf58-4704-8dbc-2cf4c3274efa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:34.078532 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.078491    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/450701f9-bf58-4704-8dbc-2cf4c3274efa-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:44:34.078532 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.078527    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/450701f9-bf58-4704-8dbc-2cf4c3274efa-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:44:34.218617 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.218575    2583 generic.go:356] "Generic (PLEG): container finished" podID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerID="71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1" exitCode=0
Sep 17 23:44:34.218795 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.218633    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"
Sep 17 23:44:34.218795 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.218667    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" event={"ID":"450701f9-bf58-4704-8dbc-2cf4c3274efa","Type":"ContainerDied","Data":"71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1"}
Sep 17 23:44:34.218795 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.218725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p" event={"ID":"450701f9-bf58-4704-8dbc-2cf4c3274efa","Type":"ContainerDied","Data":"b6b9ab55c702f345643c2d9504847fbdf7e2533d754ea80adf6b5f1f1d06dabd"}
Sep 17 23:44:34.218795 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.218750    2583 scope.go:117] "RemoveContainer" containerID="71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1"
Sep 17 23:44:34.228568 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.228547    2583 scope.go:117] "RemoveContainer" containerID="71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1"
Sep 17 23:44:34.228964 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:44:34.228940    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1\": container with ID starting with 71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1 not found: ID does not exist" containerID="71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1"
Sep 17 23:44:34.229073 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.228974    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1"} err="failed to get container status \"71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1\": rpc error: code = NotFound desc = could not find container \"71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1\": container with ID starting with 71e0496e4d311a419d0deda19801dde52a225b26a5d600d73681855ba2bfcfe1 not found: ID does not exist"
Sep 17 23:44:34.247716 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.247676    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"]
Sep 17 23:44:34.251250 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:34.251216    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fc189-8457d9b884-rsm8p"]
Sep 17 23:44:35.157533 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:35.157493    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" path="/var/lib/kubelet/pods/450701f9-bf58-4704-8dbc-2cf4c3274efa/volumes"
Sep 17 23:44:40.019372 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019333    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"]
Sep 17 23:44:40.019816 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019710    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189"
Sep 17 23:44:40.019816 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019724    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189"
Sep 17 23:44:40.019816 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019737    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kube-rbac-proxy"
Sep 17 23:44:40.019816 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019744    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kube-rbac-proxy"
Sep 17 23:44:40.019816 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019766    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container"
Sep 17 23:44:40.019816 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019774    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container"
Sep 17 23:44:40.020014 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019830    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kserve-container"
Sep 17 23:44:40.020014 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019840    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="450701f9-bf58-4704-8dbc-2cf4c3274efa" containerName="splitter-graph-fc189"
Sep 17 23:44:40.020014 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.019852    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4f4541d-671a-4d17-87dc-a4a7471d5268" containerName="kube-rbac-proxy"
Sep 17 23:44:40.024395 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.024368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:40.026964 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.026932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d4492c4-73ec-4ea0-b22f-76077549f5c3-proxy-tls\") pod \"switch-graph-57566-869f5ff548-h6h58\" (UID: \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\") " pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:40.027108 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.026977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d4492c4-73ec-4ea0-b22f-76077549f5c3-openshift-service-ca-bundle\") pod \"switch-graph-57566-869f5ff548-h6h58\" (UID: \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\") " pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:40.027200 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.027184    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-57566-serving-cert\""
Sep 17 23:44:40.027238 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.027184    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-57566-kube-rbac-proxy-sar-config\""
Sep 17 23:44:40.033106 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.033078    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"]
Sep 17 23:44:40.127798 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.127761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d4492c4-73ec-4ea0-b22f-76077549f5c3-proxy-tls\") pod \"switch-graph-57566-869f5ff548-h6h58\" (UID: \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\") " pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:40.127973 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.127805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d4492c4-73ec-4ea0-b22f-76077549f5c3-openshift-service-ca-bundle\") pod \"switch-graph-57566-869f5ff548-h6h58\" (UID: \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\") " pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:40.127973 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:44:40.127937    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-57566-serving-cert: secret "switch-graph-57566-serving-cert" not found
Sep 17 23:44:40.128072 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:44:40.128007    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4492c4-73ec-4ea0-b22f-76077549f5c3-proxy-tls podName:8d4492c4-73ec-4ea0-b22f-76077549f5c3 nodeName:}" failed. No retries permitted until 2026-09-17 23:44:40.627991254 +0000 UTC m=+2043.562436983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8d4492c4-73ec-4ea0-b22f-76077549f5c3-proxy-tls") pod "switch-graph-57566-869f5ff548-h6h58" (UID: "8d4492c4-73ec-4ea0-b22f-76077549f5c3") : secret "switch-graph-57566-serving-cert" not found
Sep 17 23:44:40.128423 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.128406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d4492c4-73ec-4ea0-b22f-76077549f5c3-openshift-service-ca-bundle\") pod \"switch-graph-57566-869f5ff548-h6h58\" (UID: \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\") " pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:40.631137 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.631092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d4492c4-73ec-4ea0-b22f-76077549f5c3-proxy-tls\") pod \"switch-graph-57566-869f5ff548-h6h58\" (UID: \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\") " pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:40.633781 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.633740    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d4492c4-73ec-4ea0-b22f-76077549f5c3-proxy-tls\") pod \"switch-graph-57566-869f5ff548-h6h58\" (UID: \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\") " pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:40.635499 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.635477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:40.913133 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:40.913044    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"]
Sep 17 23:44:40.916440 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:44:40.916398    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4492c4_73ec_4ea0_b22f_76077549f5c3.slice/crio-a99bb0da9ad4e0cfb66287891c66009aaa52d476f77b5b2d15c4e179d807963b WatchSource:0}: Error finding container a99bb0da9ad4e0cfb66287891c66009aaa52d476f77b5b2d15c4e179d807963b: Status 404 returned error can't find the container with id a99bb0da9ad4e0cfb66287891c66009aaa52d476f77b5b2d15c4e179d807963b
Sep 17 23:44:41.253283 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:41.253184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" event={"ID":"8d4492c4-73ec-4ea0-b22f-76077549f5c3","Type":"ContainerStarted","Data":"29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971"}
Sep 17 23:44:41.253283 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:41.253252    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" event={"ID":"8d4492c4-73ec-4ea0-b22f-76077549f5c3","Type":"ContainerStarted","Data":"a99bb0da9ad4e0cfb66287891c66009aaa52d476f77b5b2d15c4e179d807963b"}
Sep 17 23:44:41.253693 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:41.253384    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:41.274252 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:41.274192    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" podStartSLOduration=1.274174768 podStartE2EDuration="1.274174768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:41.271820632 +0000 UTC m=+2044.206266382" watchObservedRunningTime="2026-09-17 23:44:41.274174768 +0000 UTC m=+2044.208620518"
Sep 17 23:44:42.093375 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:42.093333    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 23:44:47.264153 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:47.264115    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 17 23:44:52.093470 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:44:52.093372    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 23:45:02.094970 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:02.094933    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:45:13.909104 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:13.909063    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"]
Sep 17 23:45:13.912538 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:13.912519    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:13.914826 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:13.914797    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-daa97-kube-rbac-proxy-sar-config\""
Sep 17 23:45:13.915014 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:13.914993    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-daa97-serving-cert\""
Sep 17 23:45:13.925447 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:13.925416    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"]
Sep 17 23:45:13.974231 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:13.974194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be8634a-66b8-4a7b-b457-394c532d9e71-proxy-tls\") pod \"splitter-graph-daa97-dfb866868-gpr4r\" (UID: \"3be8634a-66b8-4a7b-b457-394c532d9e71\") " pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:13.974453 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:13.974284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be8634a-66b8-4a7b-b457-394c532d9e71-openshift-service-ca-bundle\") pod \"splitter-graph-daa97-dfb866868-gpr4r\" (UID: \"3be8634a-66b8-4a7b-b457-394c532d9e71\") " pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:14.074742 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:14.074704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be8634a-66b8-4a7b-b457-394c532d9e71-openshift-service-ca-bundle\") pod \"splitter-graph-daa97-dfb866868-gpr4r\" (UID: \"3be8634a-66b8-4a7b-b457-394c532d9e71\") " pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:14.074922 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:14.074797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be8634a-66b8-4a7b-b457-394c532d9e71-proxy-tls\") pod \"splitter-graph-daa97-dfb866868-gpr4r\" (UID: \"3be8634a-66b8-4a7b-b457-394c532d9e71\") " pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:14.074922 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:45:14.074911    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-daa97-serving-cert: secret "splitter-graph-daa97-serving-cert" not found
Sep 17 23:45:14.075104 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:45:14.074994    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be8634a-66b8-4a7b-b457-394c532d9e71-proxy-tls podName:3be8634a-66b8-4a7b-b457-394c532d9e71 nodeName:}" failed. No retries permitted until 2026-09-17 23:45:14.574978649 +0000 UTC m=+2077.509424390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3be8634a-66b8-4a7b-b457-394c532d9e71-proxy-tls") pod "splitter-graph-daa97-dfb866868-gpr4r" (UID: "3be8634a-66b8-4a7b-b457-394c532d9e71") : secret "splitter-graph-daa97-serving-cert" not found
Sep 17 23:45:14.075378 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:14.075359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be8634a-66b8-4a7b-b457-394c532d9e71-openshift-service-ca-bundle\") pod \"splitter-graph-daa97-dfb866868-gpr4r\" (UID: \"3be8634a-66b8-4a7b-b457-394c532d9e71\") " pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:14.578800 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:14.578764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be8634a-66b8-4a7b-b457-394c532d9e71-proxy-tls\") pod \"splitter-graph-daa97-dfb866868-gpr4r\" (UID: \"3be8634a-66b8-4a7b-b457-394c532d9e71\") " pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:14.581388 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:14.581359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be8634a-66b8-4a7b-b457-394c532d9e71-proxy-tls\") pod \"splitter-graph-daa97-dfb866868-gpr4r\" (UID: \"3be8634a-66b8-4a7b-b457-394c532d9e71\") " pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:14.823767 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:14.823720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:15.099772 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:15.099598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"]
Sep 17 23:45:15.102500 ip-10-0-142-96 kubenswrapper[2583]: W0917 23:45:15.102469    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be8634a_66b8_4a7b_b457_394c532d9e71.slice/crio-17c4f9472bd0f45de1173b23aed88ee6ea7cbcca93193dc4e285a524a6c70381 WatchSource:0}: Error finding container 17c4f9472bd0f45de1173b23aed88ee6ea7cbcca93193dc4e285a524a6c70381: Status 404 returned error can't find the container with id 17c4f9472bd0f45de1173b23aed88ee6ea7cbcca93193dc4e285a524a6c70381
Sep 17 23:45:15.417848 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:15.417758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" event={"ID":"3be8634a-66b8-4a7b-b457-394c532d9e71","Type":"ContainerStarted","Data":"3b6e600d3eae4edd7cae1dfdd8f4cc0819941fd773add89f0cbeef30a1fb1246"}
Sep 17 23:45:15.417848 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:15.417815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" event={"ID":"3be8634a-66b8-4a7b-b457-394c532d9e71","Type":"ContainerStarted","Data":"17c4f9472bd0f45de1173b23aed88ee6ea7cbcca93193dc4e285a524a6c70381"}
Sep 17 23:45:15.418064 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:15.417891    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:15.436271 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:15.436214    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" podStartSLOduration=2.436195214 podStartE2EDuration="2.436195214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:15.435391842 +0000 UTC m=+2078.369837591" watchObservedRunningTime="2026-09-17 23:45:15.436195214 +0000 UTC m=+2078.370640962"
Sep 17 23:45:21.428305 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:21.428265    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:45:39.228551 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:39.228520    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:45:39.231662 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:39.231635    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:45:39.231881 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:39.231865    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:45:39.234994 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:45:39.234971    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:50:39.255970 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:50:39.255937    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:50:39.259306 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:50:39.259277    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:50:39.260457 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:50:39.260431    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:50:39.263939 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:50:39.263918    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:53:28.548381 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:28.548336    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"]
Sep 17 23:53:28.548991 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:28.548609    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97" containerID="cri-o://3b6e600d3eae4edd7cae1dfdd8f4cc0819941fd773add89f0cbeef30a1fb1246" gracePeriod=30
Sep 17 23:53:28.716409 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:28.716373    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"]
Sep 17 23:53:28.716701 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:28.716651    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container" containerID="cri-o://301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d" gracePeriod=30
Sep 17 23:53:28.716788 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:28.716664    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kube-rbac-proxy" containerID="cri-o://a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2" gracePeriod=30
Sep 17 23:53:29.708815 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:29.708780    2583 generic.go:356] "Generic (PLEG): container finished" podID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerID="a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2" exitCode=2
Sep 17 23:53:29.709242 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:29.708854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" event={"ID":"eb2de6d8-6144-4506-aac2-678e98ea3f36","Type":"ContainerDied","Data":"a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2"}
Sep 17 23:53:31.427771 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:31.427730    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:53:32.088606 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.088550    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 23:53:32.093874 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.093839    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 23:53:32.278912 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.278880    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:53:32.351080 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.350951    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnsnq\" (UniqueName: \"kubernetes.io/projected/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-api-access-mnsnq\") pod \"eb2de6d8-6144-4506-aac2-678e98ea3f36\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") "
Sep 17 23:53:32.351080 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.351016    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-rbac-proxy-sar-config\") pod \"eb2de6d8-6144-4506-aac2-678e98ea3f36\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") "
Sep 17 23:53:32.351080 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.351067    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb2de6d8-6144-4506-aac2-678e98ea3f36-proxy-tls\") pod \"eb2de6d8-6144-4506-aac2-678e98ea3f36\" (UID: \"eb2de6d8-6144-4506-aac2-678e98ea3f36\") "
Sep 17 23:53:32.351430 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.351403    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb2de6d8-6144-4506-aac2-678e98ea3f36" (UID: "eb2de6d8-6144-4506-aac2-678e98ea3f36"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:53:32.353338 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.353301    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2de6d8-6144-4506-aac2-678e98ea3f36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb2de6d8-6144-4506-aac2-678e98ea3f36" (UID: "eb2de6d8-6144-4506-aac2-678e98ea3f36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:53:32.353338 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.353323    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-api-access-mnsnq" (OuterVolumeSpecName: "kube-api-access-mnsnq") pod "eb2de6d8-6144-4506-aac2-678e98ea3f36" (UID: "eb2de6d8-6144-4506-aac2-678e98ea3f36"). InnerVolumeSpecName "kube-api-access-mnsnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:53:32.451976 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.451937    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:53:32.451976 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.451970    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb2de6d8-6144-4506-aac2-678e98ea3f36-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:53:32.451976 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.451983    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnsnq\" (UniqueName: \"kubernetes.io/projected/eb2de6d8-6144-4506-aac2-678e98ea3f36-kube-api-access-mnsnq\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:53:32.726441 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.726338    2583 generic.go:356] "Generic (PLEG): container finished" podID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerID="301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d" exitCode=0
Sep 17 23:53:32.726441 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.726397    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"
Sep 17 23:53:32.726441 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.726420    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" event={"ID":"eb2de6d8-6144-4506-aac2-678e98ea3f36","Type":"ContainerDied","Data":"301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d"}
Sep 17 23:53:32.726728 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.726480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx" event={"ID":"eb2de6d8-6144-4506-aac2-678e98ea3f36","Type":"ContainerDied","Data":"ea07f49533cd0b3d041bb7a842a285ba3f8c14176f4afdffefc99e224e20050c"}
Sep 17 23:53:32.726728 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.726515    2583 scope.go:117] "RemoveContainer" containerID="a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2"
Sep 17 23:53:32.745889 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.745866    2583 scope.go:117] "RemoveContainer" containerID="301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d"
Sep 17 23:53:32.754806 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.754776    2583 scope.go:117] "RemoveContainer" containerID="a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2"
Sep 17 23:53:32.755230 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.755207    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"]
Sep 17 23:53:32.755315 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:53:32.755279    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2\": container with ID starting with a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2 not found: ID does not exist" containerID="a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2"
Sep 17 23:53:32.755384 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.755327    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2"} err="failed to get container status \"a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2\": rpc error: code = NotFound desc = could not find container \"a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2\": container with ID starting with a59fed2ea1c8edc8b5424ffe27989e7ef2f619cc0d5f9fcef87adac2f98ba8b2 not found: ID does not exist"
Sep 17 23:53:32.755384 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.755355    2583 scope.go:117] "RemoveContainer" containerID="301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d"
Sep 17 23:53:32.755690 ip-10-0-142-96 kubenswrapper[2583]: E0917 23:53:32.755673    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d\": container with ID starting with 301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d not found: ID does not exist" containerID="301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d"
Sep 17 23:53:32.755744 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.755696    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d"} err="failed to get container status \"301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d\": rpc error: code = NotFound desc = could not find container \"301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d\": container with ID starting with 301d1d449bc4289e0ed0cbff99854f111dcf25e9998caba01ea6db7226399e5d not found: ID does not exist"
Sep 17 23:53:32.759111 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:32.759074    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daa97-predictor-845479f546-9wxxx"]
Sep 17 23:53:33.157384 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:33.157336    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" path="/var/lib/kubelet/pods/eb2de6d8-6144-4506-aac2-678e98ea3f36/volumes"
Sep 17 23:53:36.427343 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:36.427301    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:53:41.426956 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:41.426914    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:53:41.427368 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:41.427048    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:53:46.426785 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:46.426741    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:53:51.426765 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:51.426668    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:53:56.426696 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:56.426657    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:53:58.849048 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:58.848931    2583 generic.go:356] "Generic (PLEG): container finished" podID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerID="3b6e600d3eae4edd7cae1dfdd8f4cc0819941fd773add89f0cbeef30a1fb1246" exitCode=0
Sep 17 23:53:58.849048 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:58.848991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" event={"ID":"3be8634a-66b8-4a7b-b457-394c532d9e71","Type":"ContainerDied","Data":"3b6e600d3eae4edd7cae1dfdd8f4cc0819941fd773add89f0cbeef30a1fb1246"}
Sep 17 23:53:59.204358 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.204324    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:53:59.280485 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.280438    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be8634a-66b8-4a7b-b457-394c532d9e71-proxy-tls\") pod \"3be8634a-66b8-4a7b-b457-394c532d9e71\" (UID: \"3be8634a-66b8-4a7b-b457-394c532d9e71\") "
Sep 17 23:53:59.280695 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.280554    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be8634a-66b8-4a7b-b457-394c532d9e71-openshift-service-ca-bundle\") pod \"3be8634a-66b8-4a7b-b457-394c532d9e71\" (UID: \"3be8634a-66b8-4a7b-b457-394c532d9e71\") "
Sep 17 23:53:59.280936 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.280908    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be8634a-66b8-4a7b-b457-394c532d9e71-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3be8634a-66b8-4a7b-b457-394c532d9e71" (UID: "3be8634a-66b8-4a7b-b457-394c532d9e71"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:53:59.282716 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.282691    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be8634a-66b8-4a7b-b457-394c532d9e71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3be8634a-66b8-4a7b-b457-394c532d9e71" (UID: "3be8634a-66b8-4a7b-b457-394c532d9e71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:53:59.381369 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.381261    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3be8634a-66b8-4a7b-b457-394c532d9e71-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:53:59.381369 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.381313    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be8634a-66b8-4a7b-b457-394c532d9e71-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 17 23:53:59.854775 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.854741    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"
Sep 17 23:53:59.855265 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.854785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r" event={"ID":"3be8634a-66b8-4a7b-b457-394c532d9e71","Type":"ContainerDied","Data":"17c4f9472bd0f45de1173b23aed88ee6ea7cbcca93193dc4e285a524a6c70381"}
Sep 17 23:53:59.855265 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.854855    2583 scope.go:117] "RemoveContainer" containerID="3b6e600d3eae4edd7cae1dfdd8f4cc0819941fd773add89f0cbeef30a1fb1246"
Sep 17 23:53:59.876967 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.876887    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"]
Sep 17 23:53:59.878786 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:53:59.878753    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-daa97-dfb866868-gpr4r"]
Sep 17 23:54:01.157608 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:54:01.157569    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" path="/var/lib/kubelet/pods/3be8634a-66b8-4a7b-b457-394c532d9e71/volumes"
Sep 17 23:55:39.282924 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:55:39.282886    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:55:39.286185 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:55:39.286159    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 17 23:55:39.293405 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:55:39.293381    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 17 23:55:39.296207 ip-10-0-142-96 kubenswrapper[2583]: I0917 23:55:39.296187    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 18 00:00:39.311421 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:39.311288    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 18 00:00:39.317283 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:39.314799    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 18 00:00:39.321752 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:39.321730    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 18 00:00:39.324664 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:39.324644    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 18 00:00:59.363502 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:59.363453    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"]
Sep 18 00:00:59.364128 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:59.363704    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566" containerID="cri-o://29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971" gracePeriod=30
Sep 18 00:00:59.482481 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:59.482445    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"]
Sep 18 00:00:59.482710 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:59.482682    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container" containerID="cri-o://ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8" gracePeriod=30
Sep 18 00:00:59.482816 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:59.482779    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kube-rbac-proxy" containerID="cri-o://9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f" gracePeriod=30
Sep 18 00:00:59.764600 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:59.764500    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerID="9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f" exitCode=2
Sep 18 00:00:59.764600 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:00:59.764573    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" event={"ID":"1c151445-b6ed-41c4-9e32-362ee62596ed","Type":"ContainerDied","Data":"9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f"}
Sep 18 00:01:02.261896 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:02.261849    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 00:01:03.036133 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.036103    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 18 00:01:03.136544 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.136452    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-rbac-proxy-sar-config\") pod \"1c151445-b6ed-41c4-9e32-362ee62596ed\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") "
Sep 18 00:01:03.136544 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.136496    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c151445-b6ed-41c4-9e32-362ee62596ed-proxy-tls\") pod \"1c151445-b6ed-41c4-9e32-362ee62596ed\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") "
Sep 18 00:01:03.136749 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.136553    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jdpt\" (UniqueName: \"kubernetes.io/projected/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-api-access-6jdpt\") pod \"1c151445-b6ed-41c4-9e32-362ee62596ed\" (UID: \"1c151445-b6ed-41c4-9e32-362ee62596ed\") "
Sep 18 00:01:03.136967 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.136928    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c151445-b6ed-41c4-9e32-362ee62596ed" (UID: "1c151445-b6ed-41c4-9e32-362ee62596ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:01:03.138784 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.138751    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-api-access-6jdpt" (OuterVolumeSpecName: "kube-api-access-6jdpt") pod "1c151445-b6ed-41c4-9e32-362ee62596ed" (UID: "1c151445-b6ed-41c4-9e32-362ee62596ed"). InnerVolumeSpecName "kube-api-access-6jdpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:01:03.138784 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.138763    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c151445-b6ed-41c4-9e32-362ee62596ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c151445-b6ed-41c4-9e32-362ee62596ed" (UID: "1c151445-b6ed-41c4-9e32-362ee62596ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:01:03.237887 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.237845    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 18 00:01:03.237887 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.237879    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c151445-b6ed-41c4-9e32-362ee62596ed-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 18 00:01:03.237887 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.237896    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jdpt\" (UniqueName: \"kubernetes.io/projected/1c151445-b6ed-41c4-9e32-362ee62596ed-kube-api-access-6jdpt\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 18 00:01:03.785428 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.785380    2583 generic.go:356] "Generic (PLEG): container finished" podID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerID="ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8" exitCode=0
Sep 18 00:01:03.785902 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.785448    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"
Sep 18 00:01:03.785902 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.785470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" event={"ID":"1c151445-b6ed-41c4-9e32-362ee62596ed","Type":"ContainerDied","Data":"ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8"}
Sep 18 00:01:03.785902 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.785564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7" event={"ID":"1c151445-b6ed-41c4-9e32-362ee62596ed","Type":"ContainerDied","Data":"9e57f59fc88a0ea1f2e308ea4d7cdd5b043c84a695b360fcc61902cd61c29194"}
Sep 18 00:01:03.785902 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.785592    2583 scope.go:117] "RemoveContainer" containerID="9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f"
Sep 18 00:01:03.794437 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.794413    2583 scope.go:117] "RemoveContainer" containerID="ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8"
Sep 18 00:01:03.802419 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.802385    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"]
Sep 18 00:01:03.802879 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.802860    2583 scope.go:117] "RemoveContainer" containerID="9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f"
Sep 18 00:01:03.803235 ip-10-0-142-96 kubenswrapper[2583]: E0918 00:01:03.803212    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f\": container with ID starting with 9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f not found: ID does not exist" containerID="9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f"
Sep 18 00:01:03.803309 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.803243    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f"} err="failed to get container status \"9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f\": rpc error: code = NotFound desc = could not find container \"9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f\": container with ID starting with 9188607c82e3e23f665a4dde0282c8c6f2a03a644ead8438c13de8c69348585f not found: ID does not exist"
Sep 18 00:01:03.803309 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.803264    2583 scope.go:117] "RemoveContainer" containerID="ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8"
Sep 18 00:01:03.803520 ip-10-0-142-96 kubenswrapper[2583]: E0918 00:01:03.803502    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8\": container with ID starting with ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8 not found: ID does not exist" containerID="ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8"
Sep 18 00:01:03.803559 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.803529    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8"} err="failed to get container status \"ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8\": rpc error: code = NotFound desc = could not find container \"ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8\": container with ID starting with ede081734c7a0969ea0f465fa7b90a6ab7b40131ed66e0a217ff1e30add01af8 not found: ID does not exist"
Sep 18 00:01:03.805928 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:03.805903    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-57566-predictor-6949f9f8f6-9bgg7"]
Sep 18 00:01:05.157453 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:05.157416    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" path="/var/lib/kubelet/pods/1c151445-b6ed-41c4-9e32-362ee62596ed/volumes"
Sep 18 00:01:07.262343 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:07.262299    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 00:01:12.262334 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:12.262290    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 00:01:12.262734 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:12.262411    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 18 00:01:16.935331 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:16.935290    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:17.262098 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:17.261988    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 00:01:17.793726 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:17.793680    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:18.605398 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:18.605362    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:19.396072 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:19.396039    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:20.183010 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:20.182976    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:20.969199 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:20.969172    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:21.752932 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:21.752886    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:22.262152 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:22.262106    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 00:01:22.548478 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:22.548449    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:23.313738 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:23.313699    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:24.114685 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:24.114646    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:24.892842 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:24.892789    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:25.690717 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:25.690652    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:26.525226 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:26.525190    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:27.261839 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:27.261786    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 00:01:27.356351 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:27.356321    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:28.247719 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:28.247687    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-57566-869f5ff548-h6h58_8d4492c4-73ec-4ea0-b22f-76077549f5c3/switch-graph-57566/0.log"
Sep 18 00:01:29.523744 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.523716    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 18 00:01:29.540848 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.540812    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d4492c4-73ec-4ea0-b22f-76077549f5c3-proxy-tls\") pod \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\" (UID: \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\") "
Sep 18 00:01:29.541079 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.540870    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d4492c4-73ec-4ea0-b22f-76077549f5c3-openshift-service-ca-bundle\") pod \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\" (UID: \"8d4492c4-73ec-4ea0-b22f-76077549f5c3\") "
Sep 18 00:01:29.541584 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.541546    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4492c4-73ec-4ea0-b22f-76077549f5c3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8d4492c4-73ec-4ea0-b22f-76077549f5c3" (UID: "8d4492c4-73ec-4ea0-b22f-76077549f5c3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:01:29.543665 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.543628    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4492c4-73ec-4ea0-b22f-76077549f5c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d4492c4-73ec-4ea0-b22f-76077549f5c3" (UID: "8d4492c4-73ec-4ea0-b22f-76077549f5c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:01:29.642242 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.642145    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d4492c4-73ec-4ea0-b22f-76077549f5c3-proxy-tls\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 18 00:01:29.642242 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.642180    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d4492c4-73ec-4ea0-b22f-76077549f5c3-openshift-service-ca-bundle\") on node \"ip-10-0-142-96.ec2.internal\" DevicePath \"\""
Sep 18 00:01:29.902581 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.902480    2583 generic.go:356] "Generic (PLEG): container finished" podID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerID="29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971" exitCode=0
Sep 18 00:01:29.902581 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.902531    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"
Sep 18 00:01:29.902777 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.902575    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" event={"ID":"8d4492c4-73ec-4ea0-b22f-76077549f5c3","Type":"ContainerDied","Data":"29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971"}
Sep 18 00:01:29.902777 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.902631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58" event={"ID":"8d4492c4-73ec-4ea0-b22f-76077549f5c3","Type":"ContainerDied","Data":"a99bb0da9ad4e0cfb66287891c66009aaa52d476f77b5b2d15c4e179d807963b"}
Sep 18 00:01:29.902777 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.902655    2583 scope.go:117] "RemoveContainer" containerID="29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971"
Sep 18 00:01:29.911786 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.911763    2583 scope.go:117] "RemoveContainer" containerID="29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971"
Sep 18 00:01:29.912150 ip-10-0-142-96 kubenswrapper[2583]: E0918 00:01:29.912123    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971\": container with ID starting with 29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971 not found: ID does not exist" containerID="29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971"
Sep 18 00:01:29.912263 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.912157    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971"} err="failed to get container status \"29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971\": rpc error: code = NotFound desc = could not find container \"29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971\": container with ID starting with 29f51ed5f7d0587abba56a4d78cc0ed3f2e7824337edb60915a41f871db7a971 not found: ID does not exist"
Sep 18 00:01:29.922802 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.922766    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"]
Sep 18 00:01:29.926431 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:29.926400    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-57566-869f5ff548-h6h58"]
Sep 18 00:01:31.156971 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:31.156933    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" path="/var/lib/kubelet/pods/8d4492c4-73ec-4ea0-b22f-76077549f5c3/volumes"
Sep 18 00:01:33.681125 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:33.681077    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pzttd_986816f6-1fb8-429d-a4aa-c4778d601b56/global-pull-secret-syncer/0.log"
Sep 18 00:01:33.861706 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:33.861659    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xcwf9_b4e89f46-8753-4be5-be61-9ccbaf154c6f/konnectivity-agent/0.log"
Sep 18 00:01:33.942698 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:33.942586    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-96.ec2.internal_c587c9980c41ab67e0f45c632018ea28/haproxy/0.log"
Sep 18 00:01:38.521848 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:38.521806    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-6jkk4_0fc97855-954d-4778-b07e-b85cbdaa4868/monitoring-plugin/0.log"
Sep 18 00:01:38.552633 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:38.552603    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4tmfn_433b4c61-62dc-4448-b9a6-4ac3b681daf0/node-exporter/0.log"
Sep 18 00:01:38.574998 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:38.574968    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4tmfn_433b4c61-62dc-4448-b9a6-4ac3b681daf0/kube-rbac-proxy/0.log"
Sep 18 00:01:38.596013 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:38.595984    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4tmfn_433b4c61-62dc-4448-b9a6-4ac3b681daf0/init-textfile/0.log"
Sep 18 00:01:38.777322 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:38.777214    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-d787c_e672a6c1-77b7-4e0a-b06d-46622f97fe40/kube-rbac-proxy-main/0.log"
Sep 18 00:01:38.801068 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:38.801016    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-d787c_e672a6c1-77b7-4e0a-b06d-46622f97fe40/kube-rbac-proxy-self/0.log"
Sep 18 00:01:38.824992 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:38.824963    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-d787c_e672a6c1-77b7-4e0a-b06d-46622f97fe40/openshift-state-metrics/0.log"
Sep 18 00:01:39.019255 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:39.019219    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-mqbm8_a679c3fe-edd5-413c-9367-fee3c1a11cac/prometheus-operator/0.log"
Sep 18 00:01:39.041487 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:39.041405    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-mqbm8_a679c3fe-edd5-413c-9367-fee3c1a11cac/kube-rbac-proxy/0.log"
Sep 18 00:01:41.064344 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:41.064312    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/2.log"
Sep 18 00:01:41.072115 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:41.072055    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-g5c6q_a31ae514-8497-4e88-89e7-fa894e336f26/console-operator/3.log"
Sep 18 00:01:42.400116 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400078    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"]
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400384    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kube-rbac-proxy"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400394    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kube-rbac-proxy"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400411    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400416    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400425    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400431    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400438    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400444    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400455    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400460    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400468    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kube-rbac-proxy"
Sep 18 00:01:42.400510 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400473    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kube-rbac-proxy"
Sep 18 00:01:42.400972 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400521    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d4492c4-73ec-4ea0-b22f-76077549f5c3" containerName="switch-graph-57566"
Sep 18 00:01:42.400972 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400529    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kube-rbac-proxy"
Sep 18 00:01:42.400972 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400541    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3be8634a-66b8-4a7b-b457-394c532d9e71" containerName="splitter-graph-daa97"
Sep 18 00:01:42.400972 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400546    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c151445-b6ed-41c4-9e32-362ee62596ed" containerName="kserve-container"
Sep 18 00:01:42.400972 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400552    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kube-rbac-proxy"
Sep 18 00:01:42.400972 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.400558    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2de6d8-6144-4506-aac2-678e98ea3f36" containerName="kserve-container"
Sep 18 00:01:42.403502 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.403483    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.406012 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.405988    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b94j\"/\"openshift-service-ca.crt\""
Sep 18 00:01:42.407137 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.407117    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5b94j\"/\"default-dockercfg-zkk62\""
Sep 18 00:01:42.407252 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.407136    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b94j\"/\"kube-root-ca.crt\""
Sep 18 00:01:42.410664 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.410259    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"]
Sep 18 00:01:42.446631 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.446589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-sys\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.446824 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.446640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-lib-modules\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.446824 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.446698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6vv\" (UniqueName: \"kubernetes.io/projected/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-kube-api-access-8l6vv\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.446824 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.446723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-podres\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.446824 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.446742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-proc\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.547541 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.547502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-podres\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.547541 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.547544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-proc\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.547780 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.547615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-sys\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.547780 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.547638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-lib-modules\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.547780 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.547673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6vv\" (UniqueName: \"kubernetes.io/projected/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-kube-api-access-8l6vv\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.547780 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.547687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-podres\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.547780 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.547707    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-sys\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.547780 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.547710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-proc\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.547997 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.547805    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-lib-modules\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.555838 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.555802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6vv\" (UniqueName: \"kubernetes.io/projected/0564936d-3e2b-4a50-9b47-a1a8926f4f5d-kube-api-access-8l6vv\") pod \"perf-node-gather-daemonset-ltqng\" (UID: \"0564936d-3e2b-4a50-9b47-a1a8926f4f5d\") " pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.715210 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.715111    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:42.944354 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.944324    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-548p9_f200acdd-f4e1-4afa-a742-4d5981e022cb/dns/0.log"
Sep 18 00:01:42.963554 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.963525    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-548p9_f200acdd-f4e1-4afa-a742-4d5981e022cb/kube-rbac-proxy/0.log"
Sep 18 00:01:42.987987 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:42.987895    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"]
Sep 18 00:01:42.990908 ip-10-0-142-96 kubenswrapper[2583]: W0918 00:01:42.990877    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0564936d_3e2b_4a50_9b47_a1a8926f4f5d.slice/crio-df86a8355b718222870abfba9ff9e657e2f0f6474ed34b7a7cb1af534ddcdf39 WatchSource:0}: Error finding container df86a8355b718222870abfba9ff9e657e2f0f6474ed34b7a7cb1af534ddcdf39: Status 404 returned error can't find the container with id df86a8355b718222870abfba9ff9e657e2f0f6474ed34b7a7cb1af534ddcdf39
Sep 18 00:01:43.182786 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:43.182752    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dlnfn_1aced041-ad96-4073-8c9a-dc28c37ea129/dns-node-resolver/0.log"
Sep 18 00:01:43.750554 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:43.750522    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x9277_fb5aa21c-1af6-47e6-b02e-1417aecdc5ad/node-ca/0.log"
Sep 18 00:01:43.971338 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:43.971293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng" event={"ID":"0564936d-3e2b-4a50-9b47-a1a8926f4f5d","Type":"ContainerStarted","Data":"0344d7dd31bef1aeb08132f4164e9d5abd8b1141a62f2bbe02dba3c61794c1db"}
Sep 18 00:01:43.971509 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:43.971358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng" event={"ID":"0564936d-3e2b-4a50-9b47-a1a8926f4f5d","Type":"ContainerStarted","Data":"df86a8355b718222870abfba9ff9e657e2f0f6474ed34b7a7cb1af534ddcdf39"}
Sep 18 00:01:43.971509 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:43.971471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:43.987688 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:43.987625    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng" podStartSLOduration=1.987603017 podStartE2EDuration="1.987603017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:01:43.986554593 +0000 UTC m=+3066.921000345" watchObservedRunningTime="2026-09-18 00:01:43.987603017 +0000 UTC m=+3066.922048855"
Sep 18 00:01:44.470486 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:44.470456    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d6966fbcb-qhw8d_74ddd157-12ec-4990-b2c9-04382d423ae5/router/0.log"
Sep 18 00:01:44.847989 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:44.847955    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r5r48_79530f25-f88f-4dfa-8077-90c9a35211dd/serve-healthcheck-canary/0.log"
Sep 18 00:01:45.298396 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:45.298363    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m8dj7_786a604b-77c4-4ab1-995f-518694bc2b34/kube-rbac-proxy/0.log"
Sep 18 00:01:45.322108 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:45.322077    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m8dj7_786a604b-77c4-4ab1-995f-518694bc2b34/exporter/0.log"
Sep 18 00:01:45.348141 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:45.348112    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m8dj7_786a604b-77c4-4ab1-995f-518694bc2b34/extractor/0.log"
Sep 18 00:01:47.708512 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:47.708471    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5c5fcd7899-kx7cf_b11faebd-ccf3-4927-909b-8cdbd4901abf/manager/0.log"
Sep 18 00:01:47.815953 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:47.815922    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-444h2_eb656762-5df2-487b-9852-3fbca0c02574/server/0.log"
Sep 18 00:01:49.987068 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:49.987013    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5b94j/perf-node-gather-daemonset-ltqng"
Sep 18 00:01:51.920311 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:51.920282    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-qzpxb_a1c16b4a-7e52-4cd9-9580-82a108f0cb75/migrator/0.log"
Sep 18 00:01:51.940713 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:51.940681    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-qzpxb_a1c16b4a-7e52-4cd9-9580-82a108f0cb75/graceful-termination/0.log"
Sep 18 00:01:53.272414 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.272380    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7wwwj_79e79ce8-102c-4719-9637-12f50a0382c3/kube-multus/0.log"
Sep 18 00:01:53.443498 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.443464    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnrdw_99314a8b-f4ea-4521-ad1f-6d152fd397fb/kube-multus-additional-cni-plugins/0.log"
Sep 18 00:01:53.466362 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.466324    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnrdw_99314a8b-f4ea-4521-ad1f-6d152fd397fb/egress-router-binary-copy/0.log"
Sep 18 00:01:53.486858 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.486826    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnrdw_99314a8b-f4ea-4521-ad1f-6d152fd397fb/cni-plugins/0.log"
Sep 18 00:01:53.512527 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.512490    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnrdw_99314a8b-f4ea-4521-ad1f-6d152fd397fb/bond-cni-plugin/0.log"
Sep 18 00:01:53.537423 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.537328    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnrdw_99314a8b-f4ea-4521-ad1f-6d152fd397fb/routeoverride-cni/0.log"
Sep 18 00:01:53.557918 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.557890    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnrdw_99314a8b-f4ea-4521-ad1f-6d152fd397fb/whereabouts-cni-bincopy/0.log"
Sep 18 00:01:53.578807 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.578773    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cnrdw_99314a8b-f4ea-4521-ad1f-6d152fd397fb/whereabouts-cni/0.log"
Sep 18 00:01:53.869121 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.868999    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pk7p2_c73226c0-ffce-43af-89c5-e7197f34364e/network-metrics-daemon/0.log"
Sep 18 00:01:53.884370 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:53.884339    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pk7p2_c73226c0-ffce-43af-89c5-e7197f34364e/kube-rbac-proxy/0.log"
Sep 18 00:01:54.991781 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:54.991750    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-controller/0.log"
Sep 18 00:01:55.007392 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:55.007362    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/1.log"
Sep 18 00:01:55.007392 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:55.007392    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovn-acl-logging/0.log"
Sep 18 00:01:55.027674 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:55.027640    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/kube-rbac-proxy-node/0.log"
Sep 18 00:01:55.048320 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:55.048279    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 00:01:55.064199 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:55.064168    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/northd/0.log"
Sep 18 00:01:55.082308 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:55.082277    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/nbdb/0.log"
Sep 18 00:01:55.102401 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:55.102360    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/sbdb/0.log"
Sep 18 00:01:55.160732 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:55.160693    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xp97h_6ee7a800-e764-412e-9826-9d242b39b039/ovnkube-controller/0.log"
Sep 18 00:01:56.156942 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:56.156897    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-rkzvj_ff234264-6281-402d-8c44-e5a685191ae3/check-endpoints/0.log"
Sep 18 00:01:56.203177 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:56.203147    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-b224x_be180008-7f40-4e3c-b31a-bba2323922ef/network-check-target-container/0.log"
Sep 18 00:01:57.181448 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:57.181418    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9fctl_d9f1f9aa-a2cd-456e-9606-fdb3218caede/iptables-alerter/0.log"
Sep 18 00:01:57.773647 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:57.773612    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-hfkwd_d2c12fff-e88e-47a7-95d5-3897d1c5133a/tuned/0.log"
Sep 18 00:01:59.591862 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:59.591827    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-685d7d68b5-wngnl_5e4ae8e6-94dc-4240-84d5-530faff388dc/cluster-samples-operator/0.log"
Sep 18 00:01:59.606948 ip-10-0-142-96 kubenswrapper[2583]: I0918 00:01:59.606921    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-685d7d68b5-wngnl_5e4ae8e6-94dc-4240-84d5-530faff388dc/cluster-samples-operator-watch/0.log"