Sep 16 18:43:25.647611 ip-10-0-128-222 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:43:25.647869 ip-10-0-128-222 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:43:25.647891 ip-10-0-128-222 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:43:25.648145 ip-10-0-128-222 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:43:35.863181 ip-10-0-128-222 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:43:35.863201 ip-10-0-128-222 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 102bc98f97ac45488c5ede1bfb769964 --
Sep 16 18:45:25.815824 ip-10-0-128-222 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:45:26.242278 ip-10-0-128-222 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:26.242278 ip-10-0-128-222 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:45:26.242278 ip-10-0-128-222 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:26.242278 ip-10-0-128-222 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:45:26.242278 ip-10-0-128-222 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:45:26.245234 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.245143    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:45:26.249964 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249940    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:26.249964 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249961    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:26.249964 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249966    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:26.249964 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249969    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:26.249964 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249972    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249977    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249981    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249984    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249987    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249990    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249992    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249995    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.249998    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250000    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250003    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250005    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250008    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250011    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250015    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250018    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250021    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250024    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250027    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250029    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:26.250178 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250032    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250034    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250037    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250039    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250041    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250044    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250047    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250049    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250058    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250061    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250064    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250067    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250069    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250072    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250076    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250080    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250083    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250085    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250088    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:26.250723 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250090    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250093    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250096    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250098    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250101    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250104    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250106    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250109    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250112    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250114    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250117    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250119    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250122    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250124    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250127    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250129    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250131    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250134    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250136    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250139    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:26.251192 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250142    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250144    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250146    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250149    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250151    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250154    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250158    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250160    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250163    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250166    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250169    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250172    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250175    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250178    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250181    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250184    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250187    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250190    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250192    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250195    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:26.251698 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250198    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250201    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250204    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250206    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250209    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250211    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250214    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250216    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250219    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250221    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250224    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250226    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250229    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250231    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250233    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250236    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250987    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250994    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.250997    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251000    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:26.252196 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251004    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251006    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251009    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251012    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251014    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251017    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251020    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251022    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251025    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251028    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251031    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251033    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251036    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251039    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251042    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251045    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251048    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251051    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251053    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251055    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:26.252706 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251058    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251061    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251063    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251066    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251069    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251071    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251074    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251076    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251078    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251083    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251087    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251090    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251094    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251097    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251099    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251102    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251105    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251108    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251111    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:26.253242 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251114    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251117    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251120    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251122    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251125    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251128    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251130    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251134    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251137    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251140    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251143    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251146    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251149    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251151    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251154    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251156    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251159    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251161    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251164    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251166    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:26.253737 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251168    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251172    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251175    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251178    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251181    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251183    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251185    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251188    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251190    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251193    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251195    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251198    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251201    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251204    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251206    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251209    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251212    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251214    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251217    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:26.254227 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251219    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251223    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251226    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251229    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251231    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251234    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251236    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251238    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251241    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251246    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251249    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251252    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251254    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251257    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251260    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251263    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251266    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251688    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251693    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251696    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:26.254718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251699    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251702    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251705    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251707    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251711    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251713    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251716    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251718    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251721    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251724    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251726    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251728    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251731    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251734    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251737    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251740    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251742    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251745    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251747    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251750    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:26.255210 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251752    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251755    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251757    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251759    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251763    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251765    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251768    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251771    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251773    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251775    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251778    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251780    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251784    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251787    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251790    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251793    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251796    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251798    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251801    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:26.255718 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251803    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251806    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251808    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251810    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251813    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251817    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251820    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251823    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251826    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251828    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251831    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251833    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251836    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251838    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251840    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251843    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251845    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251848    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251850    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:26.256188 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251853    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251855    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251858    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251860    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251863    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251866    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251868    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251870    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251873    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251875    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251878    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251880    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251882    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251886    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251888    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251891    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251894    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251896    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251901    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251904    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:26.256693 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251907    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251910    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251913    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251915    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251917    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251920    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251922    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251925    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251927    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251930    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251932    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251935    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251938    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251940    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251943    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251945    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251948    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.251950    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252024    2572 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252062    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252068    2572 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:45:26.257265 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252073    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252077    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252081    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252085    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252090    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252093    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252096    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252099    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252102    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252105    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252108    2572 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252111    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252114    2572 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252118    2572 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252121    2572 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252126    2572 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252129    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252132    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252135    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252138    2572 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252141    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252144    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252148    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:45:26.257787 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252151    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252154    2572 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252157    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252159    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252163    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252165    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252169    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252172    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252175    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252177    2572 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252180    2572 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252183    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252187    2572 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252190    2572 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252193    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252195    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252198    2572 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252202    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252205    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252208    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252211    2572 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252213    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252216    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252219    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252222    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:45:26.258373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252225    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252228    2572 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252230    2572 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252234    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252237    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252242    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252247    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252250    2572 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252253    2572 flags.go:64] FLAG: --help="false"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252256    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252259    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252262    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252265    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252269    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252272    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252275    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252278    2572 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252280    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252283    2572 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252286    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252289    2572 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252292    2572 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252294    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252297    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:45:26.259057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252300    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252303    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252305    2572 flags.go:64] FLAG: --lock-file=""
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252308    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252311    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252314    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252319    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252322    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252325    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252328    2572 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252330    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252334    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252336    2572 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252341    2572 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252348    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252351    2572 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252355    2572 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252358    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252361    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252364    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252367    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252370    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252373    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252376    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252384    2572 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:45:26.259661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252387    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252390    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252393    2572 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252396    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252402    2572 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252405    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252408    2572 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252411    2572 flags.go:64] FLAG: --port="10250"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252413    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252416    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b0d6113dfdd611d9"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252419    2572 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252422    2572 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252425    2572 flags.go:64] FLAG: --register-node="true"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252428    2572 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252432    2572 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252435    2572 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252438    2572 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252441    2572 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252444    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252447    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252450    2572 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252455    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252458    2572 flags.go:64] FLAG: --runonce="false"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252461    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252464    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:45:26.260288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252467    2572 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252470    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252473    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252475    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252478    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252481    2572 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252484    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252487    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252490    2572 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252493    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252496    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252499    2572 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252501    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252506    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252509    2572 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252527    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252531    2572 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252534    2572 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252537    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252539    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252542    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252545    2572 flags.go:64] FLAG: --v="2"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252549    2572 flags.go:64] FLAG: --version="false"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252553    2572 flags.go:64] FLAG: --vmodule=""
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252557    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:45:26.260965 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.252560    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252677    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252681    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252688    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252691    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252693    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252697    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252701    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252704    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252706    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252709    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252712    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252715    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252718    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252720    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252723    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252725    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252728    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252731    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:26.261575 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252733    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252736    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252738    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252741    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252744    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252746    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252748    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252751    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252754    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252756    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252758    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252761    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252764    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252766    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252768    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252771    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252776    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252778    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252781    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252783    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:26.262056 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252786    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252788    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252791    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252794    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252796    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252799    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252801    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252804    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252806    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252809    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252811    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252814    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252816    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252819    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252821    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252824    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252827    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252830    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252832    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:26.262598 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252834    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252837    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252839    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252842    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252844    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252847    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252850    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252852    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252855    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252860    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252862    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252865    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252867    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252870    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252872    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252875    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252877    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252880    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252883    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252885    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:26.263082 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252887    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252890    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252893    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252895    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252897    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252900    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252903    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252905    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252908    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252910    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252913    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252917    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252920    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252923    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252925    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252927    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252930    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252934    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252937    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:26.263584 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252940    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252943    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.252948    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.253682    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.261894    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.261910    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.261930    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.261936    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.261991    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.261996    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262000    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262002    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262005    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262008    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:26.264053 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262011    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262013    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262016    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262020    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262024    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262027    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262030    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262033    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262036    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262039    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262043    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262047    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262050    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262053    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262056    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262059    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262061    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262064    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262067    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:26.264409 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262070    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262072    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262075    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262077    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262081    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262084    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262087    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262089    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262091    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262094    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262096    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262099    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262102    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262107    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262109    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262111    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262114    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262117    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262119    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262122    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:26.264928 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262125    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262127    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262130    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262132    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262135    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262137    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262140    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262142    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262145    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262148    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262151    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262153    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262156    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262158    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262161    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262164    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262167    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262170    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262173    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262175    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:26.265423 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262178    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262181    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262183    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262185    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262188    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262190    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262193    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262195    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262198    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262200    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262203    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262205    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262208    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262210    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262213    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262216    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262218    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262220    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262223    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262225    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:26.265924 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262228    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262231    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262233    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262235    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262238    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262240    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262242    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262245    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262247    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262250    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262253    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262256    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262258    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262260    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.262265    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:26.266432 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262399    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262404    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262407    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262409    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262412    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262415    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262418    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262420    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262422    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262425    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262427    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262430    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262432    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262435    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262437    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262440    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262442    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262445    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262447    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262449    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:45:26.266823 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262452    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262454    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262457    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262459    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262461    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262464    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262467    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262470    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262473    2572 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262475    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262478    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262480    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262483    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262485    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262487    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262490    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262492    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262496    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262499    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262502    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:45:26.267312 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262505    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262507    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262524    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262527    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262530    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262532    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262535    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262537    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262541    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262544    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262547    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262550    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262553    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262556    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262558    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262560    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262563    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262565    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262569    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:45:26.267801 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262572    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262575    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262578    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262581    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262584    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262586    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262589    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262591    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262594    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262597    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262599    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262602    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262604    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262607    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262610    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262612    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262615    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262617    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262620    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262623    2572 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:45:26.268259 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262625    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262628    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262630    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262632    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262635    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262638    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262640    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262642    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262645    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262648    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262651    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262653    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262656    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262658    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262661    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262664    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262666    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262669    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262671    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:45:26.268755 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:26.262673    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:45:26.269220 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.262685    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:45:26.269220 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.262796    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:45:26.269220 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.265194    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:45:26.269220 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.266256    2572 server.go:1019] "Starting client certificate rotation"
Sep 16 18:45:26.269220 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.266358    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:26.269220 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.266386    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:45:26.269220 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.268959    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:26.271452 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.271435    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:45:26.283486 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.283470    2572 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:45:26.288021 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.288005    2572 log.go:25] "Validated CRI v1 image API"
Sep 16 18:45:26.288765 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.288750    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:45:26.293019 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.292993    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 90b3b26b-752d-4fd3-995b-3f1f312b654f:/dev/nvme0n1p4 aff14004-e7f5-4ace-b136-75b2884873f1:/dev/nvme0n1p3]
Sep 16 18:45:26.293083 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.293018    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:45:26.299174 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.299063    2572 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:26.296656554 +0000 UTC m=+0.367753768 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113791 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25fa2310da689efef36a6a659bcfca SystemUUID:ec25fa23-10da-689e-fef3-6a6a659bcfca BootID:102bc98f-97ac-4548-8c5e-de1bfb769964 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:bf:8d:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:bf:8d:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:cf:7a:57:09:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:45:26.299174 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.299171    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:45:26.299288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.299256    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:45:26.300162 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.300145    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:26.303025 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.303001    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:45:26.303160 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.303027    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:45:26.304027 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.304017    2572 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:45:26.304067 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.304030    2572 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:45:26.304067 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.304037    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:26.304067 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.304052    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:45:26.305277 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.305268    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:45:26.309861 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.309850    2572 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:26.309978 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.309969    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:45:26.310006 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.310001    2572 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:45:26.310042 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.310014    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:45:26.310042 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.310027    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:45:26.310042 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.310037    2572 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:45:26.310130 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.310046    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:45:26.310391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.310376    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:26.310454 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.310395    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:45:26.312724 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.312708    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:45:26.315630 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.315617    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:45:26.315679 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.315648    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:45:26.317473 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317463    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:45:26.317505 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317479    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:45:26.317505 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317485    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:45:26.317505 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317491    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:45:26.317505 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317499    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:45:26.317505 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317505    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:45:26.317649 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317536    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:45:26.317649 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317542    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:45:26.317649 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317548    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:45:26.317649 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317553    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:45:26.317649 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317567    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:45:26.317649 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.317572    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:45:26.319326 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.319313    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:45:26.319364 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.319328    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:45:26.323275 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.323261    2572 server.go:1268] "Started kubelet"
Sep 16 18:45:26.323352 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.323324    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:45:26.323427 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.323411    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:45:26.323476 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.323416    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:45:26.323549 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.323484    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:45:26.324127 ip-10-0-128-222 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:45:26.327671 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.327551    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:45:26.327907 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.327890    2572 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:45:26.328015 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.327559    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:45:26.328183 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.328145    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:45:26.330248 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.330228    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:45:26.332015 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.331995    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zvgtw"
Sep 16 18:45:26.335478 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.334410    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-222.ec2.internal.18d5e176921a48f5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-222.ec2.internal,UID:ip-10-0-128-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-222.ec2.internal,},FirstTimestamp:2026-09-16 18:45:26.323226869 +0000 UTC m=+0.394324084,LastTimestamp:2026-09-16 18:45:26.323226869 +0000 UTC m=+0.394324084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-222.ec2.internal,}"
Sep 16 18:45:26.335976 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.335959    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:26.336684 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.336667    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:45:26.336811 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.336793    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:26.337373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337349    2572 factory.go:55] Registering systemd factory
Sep 16 18:45:26.337373 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337366    2572 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:45:26.337673 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337604    2572 factory.go:153] Registering CRI-O factory
Sep 16 18:45:26.337673 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337624    2572 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:45:26.337673 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337624    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:45:26.337673 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337664    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:45:26.337673 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337676    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:45:26.337918 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337684    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:45:26.337918 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337703    2572 factory.go:103] Registering Raw factory
Sep 16 18:45:26.337918 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337719    2572 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:45:26.337918 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337746    2572 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:45:26.337918 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.337757    2572 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:45:26.339054 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.339038    2572 manager.go:349] Starting recovery of all containers
Sep 16 18:45:26.344707 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.344686    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zvgtw"
Sep 16 18:45:26.346656 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.346627    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:45:26.346762 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.346746    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:45:26.349307 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.349293    2572 manager.go:354] Recovery completed
Sep 16 18:45:26.350787 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.350767    2572 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 18:45:26.353861 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.353848    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:26.355677 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.355663    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:26.355746 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.355695    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:26.355746 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.355706    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:26.356153 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.356137    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:45:26.356153 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.356148    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:45:26.356273 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.356161    2572 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:45:26.356273 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.356183    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:26.356820 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.356806    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:26.356876 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.356831    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:26.356876 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.356841    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:26.358150 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.358069    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-222.ec2.internal.18d5e176940973ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-222.ec2.internal,UID:ip-10-0-128-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-222.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-222.ec2.internal,},FirstTimestamp:2026-09-16 18:45:26.355678158 +0000 UTC m=+0.426775375,LastTimestamp:2026-09-16 18:45:26.355678158 +0000 UTC m=+0.426775375,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-222.ec2.internal,}"
Sep 16 18:45:26.358251 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.358169    2572 policy_none.go:49] "None policy: Start"
Sep 16 18:45:26.358251 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.358185    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:45:26.358251 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.358197    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:45:26.360222 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.360209    2572 policy_none.go:47] "Start"
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.401966    2572 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.402176    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.402188    2572 server.go:87] "Starting device plugin registration server"
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.402393    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.402402    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.402532    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.402625    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.402641    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.403134    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:45:26.422922 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.403174    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:26.439932 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.439858    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:45:26.441225 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.441206    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:45:26.441322 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.441258    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:45:26.441322 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.441281    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:45:26.441404 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.441359    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:45:26.443746 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.443726    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:26.502784 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.502742    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:26.503867 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.503850    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:26.503952 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.503880    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:26.503952 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.503891    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:26.503952 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.503916    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.512024 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.512010    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.512070 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.512032    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-222.ec2.internal\": node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:26.526653 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.526631    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:26.542321 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.542298    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"]
Sep 16 18:45:26.542379 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.542361    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:26.543279 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.543263    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:26.543344 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.543297    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:26.543344 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.543310    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:26.545678 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.545666    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:26.545795 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.545783    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.545829 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.545815    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:26.547528 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.547489    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:26.547528 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.547498    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:26.547666 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.547533    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:26.547666 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.547539    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:26.547666 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.547544    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:26.547666 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.547551    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:26.549840 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.549827    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.549896 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.549854    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:45:26.550563 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.550546    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:45:26.550661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.550578    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:45:26.550661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.550594    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:45:26.574168 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.574148    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-222.ec2.internal\" not found" node="ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.578627 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.578611    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-222.ec2.internal\" not found" node="ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.627461 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.627435    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:26.639610 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.639586    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c1b6418d61b672718063704460a13f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal\" (UID: \"a3c1b6418d61b672718063704460a13f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.639677 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.639614    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/254bd750f9fd800e2c1818cb4a5d50a1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-222.ec2.internal\" (UID: \"254bd750f9fd800e2c1818cb4a5d50a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.639677 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.639631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3c1b6418d61b672718063704460a13f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal\" (UID: \"a3c1b6418d61b672718063704460a13f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.728365 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.728338    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:26.740785 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.740748    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3c1b6418d61b672718063704460a13f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal\" (UID: \"a3c1b6418d61b672718063704460a13f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.740845 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.740810    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c1b6418d61b672718063704460a13f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal\" (UID: \"a3c1b6418d61b672718063704460a13f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.740845 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.740836    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/254bd750f9fd800e2c1818cb4a5d50a1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-222.ec2.internal\" (UID: \"254bd750f9fd800e2c1818cb4a5d50a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.740920 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.740865    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3c1b6418d61b672718063704460a13f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal\" (UID: \"a3c1b6418d61b672718063704460a13f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.740920 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.740874    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/254bd750f9fd800e2c1818cb4a5d50a1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-222.ec2.internal\" (UID: \"254bd750f9fd800e2c1818cb4a5d50a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.740996 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.740918    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c1b6418d61b672718063704460a13f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal\" (UID: \"a3c1b6418d61b672718063704460a13f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.829333 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.829291    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:26.876920 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.876889    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.882053 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:26.881688    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:26.930400 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:26.930369    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:27.031039 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:27.030959    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:27.131591 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:27.131542    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:27.232040 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:27.232009    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:27.265531 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.265492    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:45:27.266069 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.265646    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:27.332111 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:27.332060    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-222.ec2.internal\" not found"
Sep 16 18:45:27.334127 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.334108    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:27.336692 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.336675    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:27.337123 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.337105    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:27.346481 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.346461    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:27.347831 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.347816    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal"
Sep 16 18:45:27.348759 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.348738    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:27 +0000 UTC" deadline="2028-06-20 00:04:44.403030288 +0000 UTC"
Sep 16 18:45:27.348832 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.348759    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15413h19m17.054273582s"
Sep 16 18:45:27.348897 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.348862    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:27.357593 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.357574    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:27.376684 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.376660    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kf9j4"
Sep 16 18:45:27.384069 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.384045    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kf9j4"
Sep 16 18:45:27.495988 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:27.495952    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254bd750f9fd800e2c1818cb4a5d50a1.slice/crio-29935aab42cfd1739ff5c281dd74c401b3a239d9a6ab482cb06b8280c8b752a8 WatchSource:0}: Error finding container 29935aab42cfd1739ff5c281dd74c401b3a239d9a6ab482cb06b8280c8b752a8: Status 404 returned error can't find the container with id 29935aab42cfd1739ff5c281dd74c401b3a239d9a6ab482cb06b8280c8b752a8
Sep 16 18:45:27.496228 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:27.496206    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c1b6418d61b672718063704460a13f.slice/crio-d62da36cdce6377eb8edc42e6c85b6c17262d47ff7c4b4b7b25413838f306eee WatchSource:0}: Error finding container d62da36cdce6377eb8edc42e6c85b6c17262d47ff7c4b4b7b25413838f306eee: Status 404 returned error can't find the container with id d62da36cdce6377eb8edc42e6c85b6c17262d47ff7c4b4b7b25413838f306eee
Sep 16 18:45:27.500919 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.500902    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:27.674179 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.674101    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:27.896013 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:27.895983    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:28.236371 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.236207    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:28.311107 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.310845    2572 apiserver.go:52] "Watching apiserver"
Sep 16 18:45:28.317319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.317008    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:45:28.318272 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.318239    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-xtfbc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal","openshift-network-operator/iptables-alerter-jkvjw","openshift-ovn-kubernetes/ovnkube-node-wc29d","kube-system/global-pull-secret-syncer-sll74","kube-system/konnectivity-agent-kbf8w","openshift-cluster-node-tuning-operator/tuned-hdwsl","openshift-dns/node-resolver-4fjbx","openshift-multus/multus-additional-cni-plugins-fdj6m","openshift-multus/multus-ztpjp","openshift-multus/network-metrics-daemon-856sw","openshift-network-diagnostics/network-check-target-sz5vt","kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"]
Sep 16 18:45:28.321035 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.321006    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.323351 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.323327    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:28.323481 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.323408    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:28.323638 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.323513    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:28.323731 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.323715    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fnpc8\""
Sep 16 18:45:28.323793 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.323734    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:28.326907 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.326885    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.328370 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.328348    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.329195 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.329174    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:28.329195 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.329190    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:45:28.329356 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.329335    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mgldg\""
Sep 16 18:45:28.329492 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.329474    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:28.330102 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.330081    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:45:28.331078 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.330772    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:45:28.331078 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.330779    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mv9lz\""
Sep 16 18:45:28.331078 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.331002    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:45:28.331078 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.331016    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:45:28.331297 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.331153    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:45:28.331297 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.331280    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:45:28.331812 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.331537    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.331812 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.331628    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:28.334106 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.334085    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:28.335898 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.335874    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:45:28.336241 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.336221    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rhzj9\""
Sep 16 18:45:28.336370 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.336357    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:45:28.337104 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.337086    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.339118 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.339086    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l5pkz\""
Sep 16 18:45:28.339214 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.339191    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.339526 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.339504    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:45:28.339625 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.339532    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:45:28.339625 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.339577    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:45:28.339861 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.339841    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:45:28.339935 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.339922    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:45:28.341347 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.341328    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:45:28.341498 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.341478    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:45:28.341596 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.341510    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:45:28.342012 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.341858    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vfb2p\""
Sep 16 18:45:28.342932 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.342911    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.344983 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.344964    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:45:28.345077 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.345001    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mgw2g\""
Sep 16 18:45:28.345370 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.345243    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:45:28.345791 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.345772    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.348554 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.348534    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:45:28.348667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.348650    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvp8q\""
Sep 16 18:45:28.348959 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.348940    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.350528 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350490    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-sysctl-d\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.350658 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8462v\" (UniqueName: \"kubernetes.io/projected/fbade376-cd82-4681-bcfd-7eeffa5df8f2-kube-api-access-8462v\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.350658 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350576    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2t5\" (UniqueName: \"kubernetes.io/projected/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-kube-api-access-sl2t5\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:28.350784 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350658    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-cnibin\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.350784 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350693    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.350784 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350721    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-systemd\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.350784 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350747    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ceb6ae6a-1648-41fa-8013-7637311b5dea-ovnkube-script-lib\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.350784 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350773    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-system-cni-dir\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350798    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce358ba2-d29f-4719-b826-c8726759568c-kubelet-config\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350824    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-modprobe-d\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-lib-modules\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350908    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-cni-netd\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350932    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350950    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.350986    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-systemd-units\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-node-log\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.351016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351015    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd4d4a63-f6db-45d5-8a3f-c80fc416446e-konnectivity-ca\") pod \"konnectivity-agent-kbf8w\" (UID: \"fd4d4a63-f6db-45d5-8a3f-c80fc416446e\") " pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351032    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351047    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-sys-fs\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351062    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-kubernetes\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351079    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dp8\" (UniqueName: \"kubernetes.io/projected/31f543ac-8883-46d0-a052-128752bbe6f0-kube-api-access-24dp8\") pod \"iptables-alerter-jkvjw\" (UID: \"31f543ac-8883-46d0-a052-128752bbe6f0\") " pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351093    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-run-ovn\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351110    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85zq\" (UniqueName: \"kubernetes.io/projected/ceb6ae6a-1648-41fa-8013-7637311b5dea-kube-api-access-g85zq\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351124    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-host\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351133    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cgnpg\""
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351140    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351141    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5w95\" (UniqueName: \"kubernetes.io/projected/087987de-94e6-4f62-a0ef-868e37820cf7-kube-api-access-m5w95\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351172    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-kubelet\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351187    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-etc-openvswitch\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-run-openvswitch\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351218    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpcp\" (UniqueName: \"kubernetes.io/projected/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-kube-api-access-zlpcp\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce358ba2-d29f-4719-b826-c8726759568c-dbus\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351246    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-run\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.351667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31f543ac-8883-46d0-a052-128752bbe6f0-iptables-alerter-script\") pod \"iptables-alerter-jkvjw\" (UID: \"31f543ac-8883-46d0-a052-128752bbe6f0\") " pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351275    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31f543ac-8883-46d0-a052-128752bbe6f0-host-slash\") pod \"iptables-alerter-jkvjw\" (UID: \"31f543ac-8883-46d0-a052-128752bbe6f0\") " pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351296    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-run-ovn-kubernetes\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351310    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-cni-bin\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351320    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351325    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351354    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-sysctl-conf\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351369    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/087987de-94e6-4f62-a0ef-868e37820cf7-etc-tuned\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351384    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087987de-94e6-4f62-a0ef-868e37820cf7-tmp\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351398    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ceb6ae6a-1648-41fa-8013-7637311b5dea-ovn-node-metrics-cert\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351415    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-socket-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351427    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351447    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-run-netns\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351399    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.351478    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351515    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351536    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ceb6ae6a-1648-41fa-8013-7637311b5dea-env-overrides\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351587    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.352301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.351626    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-slash\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ceb6ae6a-1648-41fa-8013-7637311b5dea-ovnkube-config\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352315    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd4d4a63-f6db-45d5-8a3f-c80fc416446e-agent-certs\") pod \"konnectivity-agent-kbf8w\" (UID: \"fd4d4a63-f6db-45d5-8a3f-c80fc416446e\") " pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352342    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-device-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352481    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-run-systemd\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352534    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-var-lib-openvswitch\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-sysconfig\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352675    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-sys\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-var-lib-kubelet\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-log-socket\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352821    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.353173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.352874    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-registration-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.386791 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.386749    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:28 +0000 UTC" deadline="2028-03-15 14:51:50.344860008 +0000 UTC"
Sep 16 18:45:28.386791 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.386787    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13100h6m21.958075505s"
Sep 16 18:45:28.439527 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.439496    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:45:28.446393 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.446343    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal" event={"ID":"254bd750f9fd800e2c1818cb4a5d50a1","Type":"ContainerStarted","Data":"29935aab42cfd1739ff5c281dd74c401b3a239d9a6ab482cb06b8280c8b752a8"}
Sep 16 18:45:28.447888 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.447857    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal" event={"ID":"a3c1b6418d61b672718063704460a13f","Type":"ContainerStarted","Data":"d62da36cdce6377eb8edc42e6c85b6c17262d47ff7c4b4b7b25413838f306eee"}
Sep 16 18:45:28.453464 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453440    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-cni-bin\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.453595 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453479    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-run-netns\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.453595 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-var-lib-kubelet\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.453595 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:28.453595 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453553    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7rt\" (UniqueName: \"kubernetes.io/projected/67557c8c-a566-40ca-b64c-8760f4fbbbe7-kube-api-access-tq7rt\") pod \"node-ca-xtfbc\" (UID: \"67557c8c-a566-40ca-b64c-8760f4fbbbe7\") " pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.453595 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453581    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-sysctl-conf\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453603    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/087987de-94e6-4f62-a0ef-868e37820cf7-etc-tuned\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087987de-94e6-4f62-a0ef-868e37820cf7-tmp\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453632    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-cni-bin\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453649    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ceb6ae6a-1648-41fa-8013-7637311b5dea-ovn-node-metrics-cert\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453667    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-socket-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453683    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-run-netns\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453699    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453730    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-cni-dir\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453775    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67557c8c-a566-40ca-b64c-8760f4fbbbe7-host\") pod \"node-ca-xtfbc\" (UID: \"67557c8c-a566-40ca-b64c-8760f4fbbbe7\") " pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453808    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ceb6ae6a-1648-41fa-8013-7637311b5dea-env-overrides\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368-tmp-dir\") pod \"node-resolver-4fjbx\" (UID: \"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368\") " pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.453884 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453887    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-run-multus-certs\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-slash\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453941    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ceb6ae6a-1648-41fa-8013-7637311b5dea-ovnkube-config\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453967    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd4d4a63-f6db-45d5-8a3f-c80fc416446e-agent-certs\") pod \"konnectivity-agent-kbf8w\" (UID: \"fd4d4a63-f6db-45d5-8a3f-c80fc416446e\") " pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.453995    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-device-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454020    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-run-systemd\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454046    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-var-lib-openvswitch\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454071    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-sysconfig\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454097    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-socket-dir-parent\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454125    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-hostroot\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454150    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67557c8c-a566-40ca-b64c-8760f4fbbbe7-serviceca\") pod \"node-ca-xtfbc\" (UID: \"67557c8c-a566-40ca-b64c-8760f4fbbbe7\") " pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454175    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454192    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454219    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-sys\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454255    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-var-lib-kubelet\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454274    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-var-lib-cni-multus\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454282    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-sys\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.454539 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-log-socket\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454321    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-log-socket\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454324    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454354    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-registration-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-system-cni-dir\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454422    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-var-lib-cni-bin\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454451    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-sysctl-d\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8462v\" (UniqueName: \"kubernetes.io/projected/fbade376-cd82-4681-bcfd-7eeffa5df8f2-kube-api-access-8462v\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454509    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2t5\" (UniqueName: \"kubernetes.io/projected/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-kube-api-access-sl2t5\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454550    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-cnibin\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454642    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-cnibin\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454682    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-conf-dir\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454715    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-systemd\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454742    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ceb6ae6a-1648-41fa-8013-7637311b5dea-ovnkube-script-lib\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454770    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454799    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-system-cni-dir\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.455426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454829    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce358ba2-d29f-4719-b826-c8726759568c-kubelet-config\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6bdp\" (UniqueName: \"kubernetes.io/projected/b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368-kube-api-access-t6bdp\") pod \"node-resolver-4fjbx\" (UID: \"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368\") " pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454886    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-etc-kubernetes\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454912    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb82j\" (UniqueName: \"kubernetes.io/projected/53a203ba-b578-4d23-be8b-de7a9208cef2-kube-api-access-cb82j\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454953    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-modprobe-d\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.454982    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-lib-modules\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455009    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-cni-netd\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455040    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455101    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455130    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-systemd-units\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455151    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-socket-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455156    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-node-log\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455199    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-node-log\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd4d4a63-f6db-45d5-8a3f-c80fc416446e-konnectivity-ca\") pod \"konnectivity-agent-kbf8w\" (UID: \"fd4d4a63-f6db-45d5-8a3f-c80fc416446e\") " pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455312    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-var-lib-openvswitch\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.456391 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455367    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-sysconfig\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455433    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455489    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-registration-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455645    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-sysctl-d\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.455729    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ceb6ae6a-1648-41fa-8013-7637311b5dea-env-overrides\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.456136    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-cnibin\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.456288    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-systemd\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.456848    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ceb6ae6a-1648-41fa-8013-7637311b5dea-ovnkube-script-lib\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.456914    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-system-cni-dir\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.456982    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce358ba2-d29f-4719-b826-c8726759568c-kubelet-config\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.457091    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-modprobe-d\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.457173    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-lib-modules\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.457215    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-cni-netd\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.457274 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.453682    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:28.457991 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.457304    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs podName:2c4cd60c-ec4c-4284-b45e-bb39c7786cf8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.957275589 +0000 UTC m=+3.028372814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs") pod "network-metrics-daemon-856sw" (UID: "2c4cd60c-ec4c-4284-b45e-bb39c7786cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:28.457991 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.457854    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-sys-fs\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.457991 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.457893    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-run-k8s-cni-cncf-io\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.457991 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.457924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-daemon-config\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.457991 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.457928    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.457991 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.457973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-kubernetes\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dp8\" (UniqueName: \"kubernetes.io/projected/31f543ac-8883-46d0-a052-128752bbe6f0-kube-api-access-24dp8\") pod \"iptables-alerter-jkvjw\" (UID: \"31f543ac-8883-46d0-a052-128752bbe6f0\") " pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458035    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-run-ovn\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458052    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-sysctl-conf\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458064    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85zq\" (UniqueName: \"kubernetes.io/projected/ceb6ae6a-1648-41fa-8013-7637311b5dea-kube-api-access-g85zq\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.456217    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458100    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53a203ba-b578-4d23-be8b-de7a9208cef2-cni-binary-copy\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458129    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-host\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.458163    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret podName:ce358ba2-d29f-4719-b826-c8726759568c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.95814399 +0000 UTC m=+3.029241198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret") pod "global-pull-secret-syncer-sll74" (UID: "ce358ba2-d29f-4719-b826-c8726759568c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458188    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5w95\" (UniqueName: \"kubernetes.io/projected/087987de-94e6-4f62-a0ef-868e37820cf7-kube-api-access-m5w95\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458195    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-host\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-kubelet\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458242    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-slash\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458277    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-systemd-units\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.458319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458290    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-sys-fs\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.459057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458380    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-etc-kubernetes\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.459057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458545    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.459057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.458718    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.459057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459008    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-etc-openvswitch\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.459057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459042    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-run-openvswitch\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459073    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpcp\" (UniqueName: \"kubernetes.io/projected/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-kube-api-access-zlpcp\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459102    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/087987de-94e6-4f62-a0ef-868e37820cf7-tmp\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459117    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/087987de-94e6-4f62-a0ef-868e37820cf7-etc-tuned\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459138    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce358ba2-d29f-4719-b826-c8726759568c-dbus\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459170    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368-hosts-file\") pod \"node-resolver-4fjbx\" (UID: \"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368\") " pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459172    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-var-lib-kubelet\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-run\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459219    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-device-dir\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31f543ac-8883-46d0-a052-128752bbe6f0-iptables-alerter-script\") pod \"iptables-alerter-jkvjw\" (UID: \"31f543ac-8883-46d0-a052-128752bbe6f0\") " pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459260    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31f543ac-8883-46d0-a052-128752bbe6f0-host-slash\") pod \"iptables-alerter-jkvjw\" (UID: \"31f543ac-8883-46d0-a052-128752bbe6f0\") " pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.459314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-run-ovn-kubernetes\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459397    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce358ba2-d29f-4719-b826-c8726759568c-dbus\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459456    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/087987de-94e6-4f62-a0ef-868e37820cf7-run\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459541    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-kubelet\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459607    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-etc-openvswitch\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459634    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd4d4a63-f6db-45d5-8a3f-c80fc416446e-konnectivity-ca\") pod \"konnectivity-agent-kbf8w\" (UID: \"fd4d4a63-f6db-45d5-8a3f-c80fc416446e\") " pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459258    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-run-openvswitch\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459652    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-run-systemd\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459695    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31f543ac-8883-46d0-a052-128752bbe6f0-host-slash\") pod \"iptables-alerter-jkvjw\" (UID: \"31f543ac-8883-46d0-a052-128752bbe6f0\") " pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459702    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fbade376-cd82-4681-bcfd-7eeffa5df8f2-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459737    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-run-netns\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459799    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-host-run-ovn-kubernetes\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.459945 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.459859    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ceb6ae6a-1648-41fa-8013-7637311b5dea-run-ovn\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.460414 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.460018    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/31f543ac-8883-46d0-a052-128752bbe6f0-iptables-alerter-script\") pod \"iptables-alerter-jkvjw\" (UID: \"31f543ac-8883-46d0-a052-128752bbe6f0\") " pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.460414 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.460144    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ceb6ae6a-1648-41fa-8013-7637311b5dea-ovnkube-config\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.460663 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.460638    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ceb6ae6a-1648-41fa-8013-7637311b5dea-ovn-node-metrics-cert\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.462549 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.462250    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.462549 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.462507    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd4d4a63-f6db-45d5-8a3f-c80fc416446e-agent-certs\") pod \"konnectivity-agent-kbf8w\" (UID: \"fd4d4a63-f6db-45d5-8a3f-c80fc416446e\") " pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:28.467328 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.467272    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8462v\" (UniqueName: \"kubernetes.io/projected/fbade376-cd82-4681-bcfd-7eeffa5df8f2-kube-api-access-8462v\") pod \"aws-ebs-csi-driver-node-vjgwc\" (UID: \"fbade376-cd82-4681-bcfd-7eeffa5df8f2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.467557 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.467535    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2t5\" (UniqueName: \"kubernetes.io/projected/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-kube-api-access-sl2t5\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:28.469008 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.468962    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5w95\" (UniqueName: \"kubernetes.io/projected/087987de-94e6-4f62-a0ef-868e37820cf7-kube-api-access-m5w95\") pod \"tuned-hdwsl\" (UID: \"087987de-94e6-4f62-a0ef-868e37820cf7\") " pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.470282 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.470238    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpcp\" (UniqueName: \"kubernetes.io/projected/fb01d6a2-e6d0-41b6-944a-7158baacf6a0-kube-api-access-zlpcp\") pod \"multus-additional-cni-plugins-fdj6m\" (UID: \"fb01d6a2-e6d0-41b6-944a-7158baacf6a0\") " pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.470698 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.470674    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85zq\" (UniqueName: \"kubernetes.io/projected/ceb6ae6a-1648-41fa-8013-7637311b5dea-kube-api-access-g85zq\") pod \"ovnkube-node-wc29d\" (UID: \"ceb6ae6a-1648-41fa-8013-7637311b5dea\") " pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.474938 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.472519    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dp8\" (UniqueName: \"kubernetes.io/projected/31f543ac-8883-46d0-a052-128752bbe6f0-kube-api-access-24dp8\") pod \"iptables-alerter-jkvjw\" (UID: \"31f543ac-8883-46d0-a052-128752bbe6f0\") " pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.560160 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560082    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-run-netns\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560160 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560117    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-var-lib-kubelet\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560160 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560143    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7rt\" (UniqueName: \"kubernetes.io/projected/67557c8c-a566-40ca-b64c-8760f4fbbbe7-kube-api-access-tq7rt\") pod \"node-ca-xtfbc\" (UID: \"67557c8c-a566-40ca-b64c-8760f4fbbbe7\") " pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560170    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-cni-dir\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560196    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67557c8c-a566-40ca-b64c-8760f4fbbbe7-host\") pod \"node-ca-xtfbc\" (UID: \"67557c8c-a566-40ca-b64c-8760f4fbbbe7\") " pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560204    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-run-netns\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560220    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-var-lib-kubelet\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368-tmp-dir\") pod \"node-resolver-4fjbx\" (UID: \"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368\") " pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560265    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-run-multus-certs\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560296    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-cni-dir\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560297    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-socket-dir-parent\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560315    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67557c8c-a566-40ca-b64c-8760f4fbbbe7-host\") pod \"node-ca-xtfbc\" (UID: \"67557c8c-a566-40ca-b64c-8760f4fbbbe7\") " pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560337    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-socket-dir-parent\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560338    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-hostroot\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560372    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-hostroot\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560375    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67557c8c-a566-40ca-b64c-8760f4fbbbe7-serviceca\") pod \"node-ca-xtfbc\" (UID: \"67557c8c-a566-40ca-b64c-8760f4fbbbe7\") " pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560372    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-run-multus-certs\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560410    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560437    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-var-lib-cni-multus\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.560449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560462    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-system-cni-dir\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560488    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-var-lib-cni-bin\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560528    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-cnibin\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560533    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368-tmp-dir\") pod \"node-resolver-4fjbx\" (UID: \"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368\") " pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560515    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-var-lib-cni-multus\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560560    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-system-cni-dir\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560567    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-var-lib-cni-bin\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-conf-dir\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560599    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-cnibin\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560603    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-conf-dir\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6bdp\" (UniqueName: \"kubernetes.io/projected/b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368-kube-api-access-t6bdp\") pod \"node-resolver-4fjbx\" (UID: \"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368\") " pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-etc-kubernetes\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560683    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb82j\" (UniqueName: \"kubernetes.io/projected/53a203ba-b578-4d23-be8b-de7a9208cef2-kube-api-access-cb82j\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560717    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-run-k8s-cni-cncf-io\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-daemon-config\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560765    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-etc-kubernetes\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560775    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53a203ba-b578-4d23-be8b-de7a9208cef2-cni-binary-copy\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560803    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/53a203ba-b578-4d23-be8b-de7a9208cef2-host-run-k8s-cni-cncf-io\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.561264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368-hosts-file\") pod \"node-resolver-4fjbx\" (UID: \"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368\") " pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.562189 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560811    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67557c8c-a566-40ca-b64c-8760f4fbbbe7-serviceca\") pod \"node-ca-xtfbc\" (UID: \"67557c8c-a566-40ca-b64c-8760f4fbbbe7\") " pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.562189 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.560881    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368-hosts-file\") pod \"node-resolver-4fjbx\" (UID: \"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368\") " pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.562189 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.561300    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/53a203ba-b578-4d23-be8b-de7a9208cef2-multus-daemon-config\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.562189 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.561326    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53a203ba-b578-4d23-be8b-de7a9208cef2-cni-binary-copy\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.567350 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.567330    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:28.567488 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.567354    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:28.567488 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.567368    2572 projected.go:196] Error preparing data for projected volume kube-api-access-m88pl for pod openshift-network-diagnostics/network-check-target-sz5vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:28.567488 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.567438    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl podName:0c903d9e-cd3d-4818-be5d-299cfd855b2e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:29.0674193 +0000 UTC m=+3.138516533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m88pl" (UniqueName: "kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl") pod "network-check-target-sz5vt" (UID: "0c903d9e-cd3d-4818-be5d-299cfd855b2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:28.570583 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.570546    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7rt\" (UniqueName: \"kubernetes.io/projected/67557c8c-a566-40ca-b64c-8760f4fbbbe7-kube-api-access-tq7rt\") pod \"node-ca-xtfbc\" (UID: \"67557c8c-a566-40ca-b64c-8760f4fbbbe7\") " pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.570730 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.570538    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6bdp\" (UniqueName: \"kubernetes.io/projected/b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368-kube-api-access-t6bdp\") pod \"node-resolver-4fjbx\" (UID: \"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368\") " pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.571513 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.571485    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb82j\" (UniqueName: \"kubernetes.io/projected/53a203ba-b578-4d23-be8b-de7a9208cef2-kube-api-access-cb82j\") pod \"multus-ztpjp\" (UID: \"53a203ba-b578-4d23-be8b-de7a9208cef2\") " pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.635595 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.635566    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hdwsl"
Sep 16 18:45:28.653060 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.653035    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jkvjw"
Sep 16 18:45:28.664194 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.664171    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:28.671045 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.671026    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xtfbc"
Sep 16 18:45:28.678862 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.678845    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:28.684527 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.684507    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fdj6m"
Sep 16 18:45:28.691463 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.691441    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc"
Sep 16 18:45:28.699368 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.699347    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fjbx"
Sep 16 18:45:28.705164 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.705146    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ztpjp"
Sep 16 18:45:28.965444 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.965411    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:28.965649 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:28.965456    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:28.965649 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.965579    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:28.965649 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.965598    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:28.965649 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.965642    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs podName:2c4cd60c-ec4c-4284-b45e-bb39c7786cf8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:29.965628994 +0000 UTC m=+4.036726198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs") pod "network-metrics-daemon-856sw" (UID: "2c4cd60c-ec4c-4284-b45e-bb39c7786cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:28.965842 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:28.965676    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret podName:ce358ba2-d29f-4719-b826-c8726759568c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:29.965657779 +0000 UTC m=+4.036754991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret") pod "global-pull-secret-syncer-sll74" (UID: "ce358ba2-d29f-4719-b826-c8726759568c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:29.167424 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:29.167383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:29.167621 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:29.167604    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:29.167689 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:29.167629    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:29.167689 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:29.167643    2572 projected.go:196] Error preparing data for projected volume kube-api-access-m88pl for pod openshift-network-diagnostics/network-check-target-sz5vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:29.167807 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:29.167711    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl podName:0c903d9e-cd3d-4818-be5d-299cfd855b2e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:30.16769088 +0000 UTC m=+4.238788098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88pl" (UniqueName: "kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl") pod "network-check-target-sz5vt" (UID: "0c903d9e-cd3d-4818-be5d-299cfd855b2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:29.387141 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:29.387091    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:28 +0000 UTC" deadline="2028-04-21 15:07:38.403874327 +0000 UTC"
Sep 16 18:45:29.387141 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:29.387137    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13988h22m9.016740283s"
Sep 16 18:45:29.514826 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:29.514786    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4d4a63_f6db_45d5_8a3f_c80fc416446e.slice/crio-76b3df9d2a0becc47b3f25eeb8ff28a80b9b0acc7d94364f4b010672c951b27a WatchSource:0}: Error finding container 76b3df9d2a0becc47b3f25eeb8ff28a80b9b0acc7d94364f4b010672c951b27a: Status 404 returned error can't find the container with id 76b3df9d2a0becc47b3f25eeb8ff28a80b9b0acc7d94364f4b010672c951b27a
Sep 16 18:45:29.516437 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:29.516409    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67557c8c_a566_40ca_b64c_8760f4fbbbe7.slice/crio-4c319524051cad9ff729ac6e4f39806716eaed5def92ecbc1b1584c40b1a6b54 WatchSource:0}: Error finding container 4c319524051cad9ff729ac6e4f39806716eaed5def92ecbc1b1584c40b1a6b54: Status 404 returned error can't find the container with id 4c319524051cad9ff729ac6e4f39806716eaed5def92ecbc1b1584c40b1a6b54
Sep 16 18:45:29.520144 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:29.520118    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a203ba_b578_4d23_be8b_de7a9208cef2.slice/crio-fc0a89fc2793526ed09c234d9788344ee448a6a0986eff43d13b2f7375a909f2 WatchSource:0}: Error finding container fc0a89fc2793526ed09c234d9788344ee448a6a0986eff43d13b2f7375a909f2: Status 404 returned error can't find the container with id fc0a89fc2793526ed09c234d9788344ee448a6a0986eff43d13b2f7375a909f2
Sep 16 18:45:29.521072 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:29.521048    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbade376_cd82_4681_bcfd_7eeffa5df8f2.slice/crio-89f3f5d59c7b6b210a18e722c3403ef8579c0e0d639a3f893f7b9c2cdb0035b4 WatchSource:0}: Error finding container 89f3f5d59c7b6b210a18e722c3403ef8579c0e0d639a3f893f7b9c2cdb0035b4: Status 404 returned error can't find the container with id 89f3f5d59c7b6b210a18e722c3403ef8579c0e0d639a3f893f7b9c2cdb0035b4
Sep 16 18:45:29.523658 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:29.523630    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb6ae6a_1648_41fa_8013_7637311b5dea.slice/crio-7a2eaaf976a024bacb3f583e363640c050b6994043d02be19f9ef0921963c4a2 WatchSource:0}: Error finding container 7a2eaaf976a024bacb3f583e363640c050b6994043d02be19f9ef0921963c4a2: Status 404 returned error can't find the container with id 7a2eaaf976a024bacb3f583e363640c050b6994043d02be19f9ef0921963c4a2
Sep 16 18:45:29.530629 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:45:29.530122    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f543ac_8883_46d0_a052_128752bbe6f0.slice/crio-560c1fcd1aab3b716538dde39d19b896170b15511d2955844ac78540884d49df WatchSource:0}: Error finding container 560c1fcd1aab3b716538dde39d19b896170b15511d2955844ac78540884d49df: Status 404 returned error can't find the container with id 560c1fcd1aab3b716538dde39d19b896170b15511d2955844ac78540884d49df
Sep 16 18:45:29.973687 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:29.973642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:29.973849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:29.973764    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:29.974048 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:29.974017    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:29.974173 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:29.974095    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs podName:2c4cd60c-ec4c-4284-b45e-bb39c7786cf8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:31.974078365 +0000 UTC m=+6.045175569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs") pod "network-metrics-daemon-856sw" (UID: "2c4cd60c-ec4c-4284-b45e-bb39c7786cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:29.974173 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:29.974017    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:29.974274 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:29.974180    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret podName:ce358ba2-d29f-4719-b826-c8726759568c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:31.974160137 +0000 UTC m=+6.045257361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret") pod "global-pull-secret-syncer-sll74" (UID: "ce358ba2-d29f-4719-b826-c8726759568c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:30.175038 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.174947    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:30.175216 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:30.175194    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:30.175288 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:30.175226    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:30.175288 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:30.175240    2572 projected.go:196] Error preparing data for projected volume kube-api-access-m88pl for pod openshift-network-diagnostics/network-check-target-sz5vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:30.175392 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:30.175306    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl podName:0c903d9e-cd3d-4818-be5d-299cfd855b2e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:32.175286941 +0000 UTC m=+6.246384166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88pl" (UniqueName: "kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl") pod "network-check-target-sz5vt" (UID: "0c903d9e-cd3d-4818-be5d-299cfd855b2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:30.445316 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.444564    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:30.445316 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:30.444737    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:30.445316 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.444811    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:30.445316 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:30.444935    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:30.445316 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.444994    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:30.445316 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:30.445065    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:30.498435 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.498395    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal" event={"ID":"254bd750f9fd800e2c1818cb4a5d50a1","Type":"ContainerStarted","Data":"79369a603f7483552472837abbdfb9e8b36a2872a8168da0f15ff052feba6ccb"}
Sep 16 18:45:30.513023 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.512736    2572 generic.go:356] "Generic (PLEG): container finished" podID="a3c1b6418d61b672718063704460a13f" containerID="308604c551092ce9a9a18acc0a3e0121ac553d0886c9f3068a9646474744527e" exitCode=0
Sep 16 18:45:30.513023 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.512823    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal" event={"ID":"a3c1b6418d61b672718063704460a13f","Type":"ContainerDied","Data":"308604c551092ce9a9a18acc0a3e0121ac553d0886c9f3068a9646474744527e"}
Sep 16 18:45:30.524341 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.524282    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-222.ec2.internal" podStartSLOduration=2.524263966 podStartE2EDuration="2.524263966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:30.523350113 +0000 UTC m=+4.594447351" watchObservedRunningTime="2026-09-16 18:45:30.524263966 +0000 UTC m=+4.595361196"
Sep 16 18:45:30.533244 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.533110    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdj6m" event={"ID":"fb01d6a2-e6d0-41b6-944a-7158baacf6a0","Type":"ContainerStarted","Data":"3720afbb86ee8ba9bff7dfe1d814a6ed7370c9d61216f939f7cc826ef0d962cb"}
Sep 16 18:45:30.540549 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.540497    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fjbx" event={"ID":"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368","Type":"ContainerStarted","Data":"5d1102f86c67430499fa5d4020c986f254a6d9c69f8baa2ba52ac92a2f1e9170"}
Sep 16 18:45:30.563693 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.563663    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc" event={"ID":"fbade376-cd82-4681-bcfd-7eeffa5df8f2","Type":"ContainerStarted","Data":"89f3f5d59c7b6b210a18e722c3403ef8579c0e0d639a3f893f7b9c2cdb0035b4"}
Sep 16 18:45:30.568823 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.568648    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ztpjp" event={"ID":"53a203ba-b578-4d23-be8b-de7a9208cef2","Type":"ContainerStarted","Data":"fc0a89fc2793526ed09c234d9788344ee448a6a0986eff43d13b2f7375a909f2"}
Sep 16 18:45:30.576658 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.576631    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xtfbc" event={"ID":"67557c8c-a566-40ca-b64c-8760f4fbbbe7","Type":"ContainerStarted","Data":"4c319524051cad9ff729ac6e4f39806716eaed5def92ecbc1b1584c40b1a6b54"}
Sep 16 18:45:30.589343 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.589315    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jkvjw" event={"ID":"31f543ac-8883-46d0-a052-128752bbe6f0","Type":"ContainerStarted","Data":"560c1fcd1aab3b716538dde39d19b896170b15511d2955844ac78540884d49df"}
Sep 16 18:45:30.605999 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.605971    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hdwsl" event={"ID":"087987de-94e6-4f62-a0ef-868e37820cf7","Type":"ContainerStarted","Data":"f067fc18743cc9effd775ce9e12665026db5b098bd66da34b51f6985a710ee2a"}
Sep 16 18:45:30.615063 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.614918    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" event={"ID":"ceb6ae6a-1648-41fa-8013-7637311b5dea","Type":"ContainerStarted","Data":"7a2eaaf976a024bacb3f583e363640c050b6994043d02be19f9ef0921963c4a2"}
Sep 16 18:45:30.628195 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:30.628168    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kbf8w" event={"ID":"fd4d4a63-f6db-45d5-8a3f-c80fc416446e","Type":"ContainerStarted","Data":"76b3df9d2a0becc47b3f25eeb8ff28a80b9b0acc7d94364f4b010672c951b27a"}
Sep 16 18:45:31.655784 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:31.655689    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal" event={"ID":"a3c1b6418d61b672718063704460a13f","Type":"ContainerStarted","Data":"4e40dbaac1bbefa89e42ed3e9e7c5177b0146dc1856201a4b307386495e8da91"}
Sep 16 18:45:31.994488 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:31.994444    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:31.994704 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:31.994508    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:31.994704 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:31.994648    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:31.994828 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:31.994719    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs podName:2c4cd60c-ec4c-4284-b45e-bb39c7786cf8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.994700055 +0000 UTC m=+10.065797277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs") pod "network-metrics-daemon-856sw" (UID: "2c4cd60c-ec4c-4284-b45e-bb39c7786cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:31.995141 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:31.995119    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:31.995252 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:31.995180    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret podName:ce358ba2-d29f-4719-b826-c8726759568c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:35.995162065 +0000 UTC m=+10.066259287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret") pod "global-pull-secret-syncer-sll74" (UID: "ce358ba2-d29f-4719-b826-c8726759568c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:32.196352 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:32.196312    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:32.196620 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:32.196542    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:32.196620 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:32.196591    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:32.196620 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:32.196609    2572 projected.go:196] Error preparing data for projected volume kube-api-access-m88pl for pod openshift-network-diagnostics/network-check-target-sz5vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:32.196804 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:32.196672    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl podName:0c903d9e-cd3d-4818-be5d-299cfd855b2e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.196652325 +0000 UTC m=+10.267749543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88pl" (UniqueName: "kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl") pod "network-check-target-sz5vt" (UID: "0c903d9e-cd3d-4818-be5d-299cfd855b2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:32.442781 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:32.442693    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:32.442956 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:32.442842    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:32.443258 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:32.443238    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:32.443362 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:32.443347    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:32.443462 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:32.443395    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:32.443462 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:32.443440    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:34.442627 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:34.442113    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:34.442627 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:34.442258    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:34.442627 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:34.442113    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:34.442627 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:34.442313    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:34.442627 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:34.442407    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:34.442627 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:34.442504    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:36.031711 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:36.030772    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:36.031711 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:36.030874    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:36.031711 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.031023    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:36.031711 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.031089    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs podName:2c4cd60c-ec4c-4284-b45e-bb39c7786cf8 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.031066842 +0000 UTC m=+18.102164046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs") pod "network-metrics-daemon-856sw" (UID: "2c4cd60c-ec4c-4284-b45e-bb39c7786cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:36.031711 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.031590    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:36.031711 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.031668    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret podName:ce358ba2-d29f-4719-b826-c8726759568c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.031647397 +0000 UTC m=+18.102744608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret") pod "global-pull-secret-syncer-sll74" (UID: "ce358ba2-d29f-4719-b826-c8726759568c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:36.231971 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:36.231928    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:36.232174 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.232161    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:36.232257 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.232187    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:36.232257 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.232202    2572 projected.go:196] Error preparing data for projected volume kube-api-access-m88pl for pod openshift-network-diagnostics/network-check-target-sz5vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:36.232367 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.232265    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl podName:0c903d9e-cd3d-4818-be5d-299cfd855b2e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.232244151 +0000 UTC m=+18.303341360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88pl" (UniqueName: "kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl") pod "network-check-target-sz5vt" (UID: "0c903d9e-cd3d-4818-be5d-299cfd855b2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:36.443288 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:36.443209    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:36.443503 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.443320    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:36.443671 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:36.443640    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:36.443794 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.443768    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:36.443854 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:36.443811    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:36.443928 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:36.443904    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:38.444097 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:38.444060    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:38.444647 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:38.444234    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:38.444732 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:38.444690    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:38.444836 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:38.444809    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:38.444901 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:38.444889    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:38.445032 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:38.444974    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:40.442467 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:40.442434    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:40.442952 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:40.442582    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:40.442952 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:40.442434    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:40.442952 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:40.442687    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:40.442952 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:40.442434    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:40.442952 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:40.442760    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:42.442108 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:42.442066    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:42.442582 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:42.442076    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:42.442582 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:42.442184    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:42.442582 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:42.442184    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:42.442582 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:42.442287    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:42.442582 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:42.442348    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:44.096899 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:44.096856    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:44.097404 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:44.096931    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:44.097404 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.097037    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:44.097404 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.097036    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:44.097404 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.097113    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret podName:ce358ba2-d29f-4719-b826-c8726759568c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:00.097092246 +0000 UTC m=+34.168189450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret") pod "global-pull-secret-syncer-sll74" (UID: "ce358ba2-d29f-4719-b826-c8726759568c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:44.097404 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.097137    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs podName:2c4cd60c-ec4c-4284-b45e-bb39c7786cf8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:00.097129654 +0000 UTC m=+34.168226855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs") pod "network-metrics-daemon-856sw" (UID: "2c4cd60c-ec4c-4284-b45e-bb39c7786cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:44.298609 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:44.298570    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:44.298773 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.298720    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:44.298773 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.298741    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:44.298773 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.298752    2572 projected.go:196] Error preparing data for projected volume kube-api-access-m88pl for pod openshift-network-diagnostics/network-check-target-sz5vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:44.298949 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.298814    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl podName:0c903d9e-cd3d-4818-be5d-299cfd855b2e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:00.298797094 +0000 UTC m=+34.369894311 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88pl" (UniqueName: "kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl") pod "network-check-target-sz5vt" (UID: "0c903d9e-cd3d-4818-be5d-299cfd855b2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:44.441775 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:44.441687    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:44.441919 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:44.441697    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:44.441919 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.441807    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:44.441919 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.441903    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:44.442028 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:44.441699    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:44.442028 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:44.442013    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:46.442746 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:46.442712    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:46.443171 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:46.442792    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:46.443171 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:46.442812    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:46.443171 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:46.442886    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:46.443171 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:46.442927    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:46.443171 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:46.442968    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:48.442142 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:48.442112    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:48.442701 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:48.442148    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:48.442701 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:48.442119    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:48.442701 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:48.442249    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:48.442701 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:48.442342    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:48.442701 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:48.442453    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:49.704615 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.704210    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb01d6a2-e6d0-41b6-944a-7158baacf6a0" containerID="9d05c903742d71d9b2f33cc88247dc59d8928019036fff5ef15e50189852c2cc" exitCode=0
Sep 16 18:45:49.705219 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.704295    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdj6m" event={"ID":"fb01d6a2-e6d0-41b6-944a-7158baacf6a0","Type":"ContainerDied","Data":"9d05c903742d71d9b2f33cc88247dc59d8928019036fff5ef15e50189852c2cc"}
Sep 16 18:45:49.706509 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.706447    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fjbx" event={"ID":"b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368","Type":"ContainerStarted","Data":"7d85bc03f102e169db598242c8194c869a5cf310345ac139dee193cf342cfa89"}
Sep 16 18:45:49.708130 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.708109    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc" event={"ID":"fbade376-cd82-4681-bcfd-7eeffa5df8f2","Type":"ContainerStarted","Data":"e3302d6a8885b301aaacec1985cfdf341d8415a49d8954843c04e3bd670753a8"}
Sep 16 18:45:49.712413 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.712369    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ztpjp" event={"ID":"53a203ba-b578-4d23-be8b-de7a9208cef2","Type":"ContainerStarted","Data":"a85b3024e6555ddfdd9b58e17f58a69e68f11ded3d23d85caa75996e16cdffdf"}
Sep 16 18:45:49.714261 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.714233    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xtfbc" event={"ID":"67557c8c-a566-40ca-b64c-8760f4fbbbe7","Type":"ContainerStarted","Data":"078c0024b58c6e76607429b2d5a1f27087065e73467d31e936b1696c21b68ae4"}
Sep 16 18:45:49.716156 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.716128    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hdwsl" event={"ID":"087987de-94e6-4f62-a0ef-868e37820cf7","Type":"ContainerStarted","Data":"2bff7135f598da1537666d062b9434be4d72c7923af6e33c28ddf1827312e42c"}
Sep 16 18:45:49.721377 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.721345    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" event={"ID":"ceb6ae6a-1648-41fa-8013-7637311b5dea","Type":"ContainerStarted","Data":"9f29f0fa4e6c130ce043f90a321ad636f87c824aaee4feb8bab6d02877a57772"}
Sep 16 18:45:49.721477 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.721413    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" event={"ID":"ceb6ae6a-1648-41fa-8013-7637311b5dea","Type":"ContainerStarted","Data":"e1a4613244fe67d434288c7b8bed8a98af25d03c0bad02502cae9b80207b174d"}
Sep 16 18:45:49.721477 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.721465    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" event={"ID":"ceb6ae6a-1648-41fa-8013-7637311b5dea","Type":"ContainerStarted","Data":"093c8940b183e1f0eab54d37513b05f88017818a6264768acba3d9e74db4d39a"}
Sep 16 18:45:49.721620 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.721533    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" event={"ID":"ceb6ae6a-1648-41fa-8013-7637311b5dea","Type":"ContainerStarted","Data":"aefb92d1aa6f73f9a823dcca73b032342aa427115f475f7bdb6a559fbb525e1d"}
Sep 16 18:45:49.721620 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.721584    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" event={"ID":"ceb6ae6a-1648-41fa-8013-7637311b5dea","Type":"ContainerStarted","Data":"07df09b328103b5c720820a0680a266df40dcfacf619456fa099c52c4362d265"}
Sep 16 18:45:49.721709 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.721631    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" event={"ID":"ceb6ae6a-1648-41fa-8013-7637311b5dea","Type":"ContainerStarted","Data":"d9a8160728e624fcc86ea7b633f8e4e23f19efa71f16acc32f454522877988c3"}
Sep 16 18:45:49.723057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.723033    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kbf8w" event={"ID":"fd4d4a63-f6db-45d5-8a3f-c80fc416446e","Type":"ContainerStarted","Data":"603836ff915911b7719b3e9143f375d2643f53e68ae3eb993ed11b96ee73f229"}
Sep 16 18:45:49.730866 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.730829    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-222.ec2.internal" podStartSLOduration=21.730817909 podStartE2EDuration="21.730817909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:31.675244538 +0000 UTC m=+5.746341767" watchObservedRunningTime="2026-09-16 18:45:49.730817909 +0000 UTC m=+23.801915132"
Sep 16 18:45:49.753135 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.753083    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hdwsl" podStartSLOduration=3.384793563 podStartE2EDuration="22.753063699s" totalImagesPullingTime="19.368270136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:49.752735428 +0000 UTC m=+23.823832651" watchObservedRunningTime="2026-09-16 18:45:49.753063699 +0000 UTC m=+23.824160960"
Sep 16 18:45:49.785668 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.785615    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kbf8w" podStartSLOduration=3.413614312 podStartE2EDuration="22.785599354s" totalImagesPullingTime="19.371985042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:49.768249251 +0000 UTC m=+23.839346475" watchObservedRunningTime="2026-09-16 18:45:49.785599354 +0000 UTC m=+23.856696580"
Sep 16 18:45:49.816352 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.816306    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4fjbx" podStartSLOduration=3.436532774 podStartE2EDuration="22.816290963s" totalImagesPullingTime="19.379758189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:49.786336632 +0000 UTC m=+23.857433856" watchObservedRunningTime="2026-09-16 18:45:49.816290963 +0000 UTC m=+23.887388186"
Sep 16 18:45:49.816488 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.816445    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ztpjp" podStartSLOduration=3.3827436730000002 podStartE2EDuration="22.816439069s" totalImagesPullingTime="19.433695396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:49.811259906 +0000 UTC m=+23.882357130" watchObservedRunningTime="2026-09-16 18:45:49.816439069 +0000 UTC m=+23.887536292"
Sep 16 18:45:49.831724 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:49.831685    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xtfbc" podStartSLOduration=3.461144925 podStartE2EDuration="22.83167042s" totalImagesPullingTime="19.370525495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:49.831297988 +0000 UTC m=+23.902395208" watchObservedRunningTime="2026-09-16 18:45:49.83167042 +0000 UTC m=+23.902767625"
Sep 16 18:45:50.441692 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:50.441665    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:50.441836 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:50.441665    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:50.441836 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:50.441768    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:50.441949 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:50.441850    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:50.441949 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:50.441665    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:50.442052 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:50.441946    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:50.490627 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:50.490601    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:45:50.727626 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:50.727591    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc" event={"ID":"fbade376-cd82-4681-bcfd-7eeffa5df8f2","Type":"ContainerStarted","Data":"673583aea533c9fd584ac1aba069ff334d6f6c9d518bd77e80b913cb99491b8d"}
Sep 16 18:45:50.729339 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:50.729307    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jkvjw" event={"ID":"31f543ac-8883-46d0-a052-128752bbe6f0","Type":"ContainerStarted","Data":"22913e5c115bb6711b71619e6afd9a3a1d3554ae15208d4218782ddc039ed9cf"}
Sep 16 18:45:50.747480 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:50.747424    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jkvjw" podStartSLOduration=4.378035674 podStartE2EDuration="23.747404718s" totalImagesPullingTime="19.369369044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:50.74666235 +0000 UTC m=+24.817759575" watchObservedRunningTime="2026-09-16 18:45:50.747404718 +0000 UTC m=+24.818501944"
Sep 16 18:45:51.417093 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:51.416977    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:50.490623788Z","UUID":"b02869b6-5086-4e1c-aed8-3fe557929c9a","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:45:51.419648 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:51.419627    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:45:51.419798 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:51.419662    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:45:51.734742 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:51.734690    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc" event={"ID":"fbade376-cd82-4681-bcfd-7eeffa5df8f2","Type":"ContainerStarted","Data":"fc585819c50203a0885e34c016a8dc4d35882bad3479d0310588aca3ced778a0"}
Sep 16 18:45:51.738751 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:51.738721    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" event={"ID":"ceb6ae6a-1648-41fa-8013-7637311b5dea","Type":"ContainerStarted","Data":"315cecf0c1e755f6ea60efa32f5ad09530852d76c04810a80d01fbbf3d885229"}
Sep 16 18:45:51.760730 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:51.760672    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjgwc" podStartSLOduration=3.205155331 podStartE2EDuration="24.760638946s" totalImagesPullingTime="21.555483615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:51.760162503 +0000 UTC m=+25.831259728" watchObservedRunningTime="2026-09-16 18:45:51.760638946 +0000 UTC m=+25.831736170"
Sep 16 18:45:52.442039 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:52.441994    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:52.442200 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:52.441994    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:52.442200 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:52.442129    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:52.442200 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:52.441998    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:52.442200 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:52.442191    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:52.442395 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:52.442294    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:53.912221 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:53.911986    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:53.912627 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:53.912610    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:54.441563 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.441540    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:54.441656 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.441581    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:54.441656 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.441540    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:54.441656 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:54.441645    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:54.441751 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:54.441715    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:54.441832 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:54.441810    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:54.747355 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.747102    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb01d6a2-e6d0-41b6-944a-7158baacf6a0" containerID="03a69d217806e4ed733be28e33b4291822476aa2c3e60149e69e7e5253fe373e" exitCode=0
Sep 16 18:45:54.747507 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.747183    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdj6m" event={"ID":"fb01d6a2-e6d0-41b6-944a-7158baacf6a0","Type":"ContainerDied","Data":"03a69d217806e4ed733be28e33b4291822476aa2c3e60149e69e7e5253fe373e"}
Sep 16 18:45:54.751396 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.751364    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" event={"ID":"ceb6ae6a-1648-41fa-8013-7637311b5dea","Type":"ContainerStarted","Data":"bf7a04b04b01077643c18dfe87539233f819bcf39aa906e1df72cb7cffb27dba"}
Sep 16 18:45:54.751658 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.751637    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:54.751768 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.751691    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:54.751768 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.751740    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:54.752682 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.752655    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kbf8w"
Sep 16 18:45:54.767747 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.767723    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:54.832116 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:54.832069    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d" podStartSLOduration=8.393705611 podStartE2EDuration="27.832054322s" totalImagesPullingTime="19.438348711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:54.831626838 +0000 UTC m=+28.902724061" watchObservedRunningTime="2026-09-16 18:45:54.832054322 +0000 UTC m=+28.903151568"
Sep 16 18:45:55.753566 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:55.753534    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:55.768300 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:55.768278    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:45:56.352404 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:56.352327    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sll74"]
Sep 16 18:45:56.352578 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:56.352484    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:56.352641 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:56.352595    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:56.356600 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:56.356575    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-856sw"]
Sep 16 18:45:56.356733 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:56.356698    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:56.356827 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:56.356809    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:56.367161 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:56.367131    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sz5vt"]
Sep 16 18:45:56.367256 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:56.367220    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:56.367310 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:56.367296    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:56.756814 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:56.756783    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb01d6a2-e6d0-41b6-944a-7158baacf6a0" containerID="56e9493074209036902ba3aee6f6f01cb7bb7c1ff2c8594735169d19f01c4c5a" exitCode=0
Sep 16 18:45:56.757375 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:56.756869    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdj6m" event={"ID":"fb01d6a2-e6d0-41b6-944a-7158baacf6a0","Type":"ContainerDied","Data":"56e9493074209036902ba3aee6f6f01cb7bb7c1ff2c8594735169d19f01c4c5a"}
Sep 16 18:45:57.441912 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:57.441881    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:57.442079 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:57.441992    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:45:58.442239 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:58.442039    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:45:58.442719 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:58.442072    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:45:58.442719 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:58.442335    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:45:58.442719 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:58.442404    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:45:58.763700 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:58.763662    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb01d6a2-e6d0-41b6-944a-7158baacf6a0" containerID="af90b574635ca21fd01c9d2abef2ad9f0523aab32624940e41f599fd2e877846" exitCode=0
Sep 16 18:45:58.763865 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:58.763705    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdj6m" event={"ID":"fb01d6a2-e6d0-41b6-944a-7158baacf6a0","Type":"ContainerDied","Data":"af90b574635ca21fd01c9d2abef2ad9f0523aab32624940e41f599fd2e877846"}
Sep 16 18:45:59.442474 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:45:59.442445    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:45:59.442917 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:45:59.442557    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:46:00.112816 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:00.112784    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:46:00.112998 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:00.112862    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:46:00.112998 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.112979    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:00.112998 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.112976    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:00.113159 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.113046    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret podName:ce358ba2-d29f-4719-b826-c8726759568c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:32.11302443 +0000 UTC m=+66.184121638 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret") pod "global-pull-secret-syncer-sll74" (UID: "ce358ba2-d29f-4719-b826-c8726759568c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:46:00.113159 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.113067    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs podName:2c4cd60c-ec4c-4284-b45e-bb39c7786cf8 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:32.113055267 +0000 UTC m=+66.184152473 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs") pod "network-metrics-daemon-856sw" (UID: "2c4cd60c-ec4c-4284-b45e-bb39c7786cf8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:46:00.314312 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:00.314273    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:46:00.314500 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.314449    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:46:00.314500 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.314473    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:46:00.314500 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.314486    2572 projected.go:196] Error preparing data for projected volume kube-api-access-m88pl for pod openshift-network-diagnostics/network-check-target-sz5vt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:00.314693 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.314559    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl podName:0c903d9e-cd3d-4818-be5d-299cfd855b2e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:32.314543413 +0000 UTC m=+66.385640619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m88pl" (UniqueName: "kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl") pod "network-check-target-sz5vt" (UID: "0c903d9e-cd3d-4818-be5d-299cfd855b2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:46:00.442504 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:00.442428    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:46:00.442930 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.442560    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz5vt" podUID="0c903d9e-cd3d-4818-be5d-299cfd855b2e"
Sep 16 18:46:00.442930 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:00.442622    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:46:00.442930 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:00.442809    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-856sw" podUID="2c4cd60c-ec4c-4284-b45e-bb39c7786cf8"
Sep 16 18:46:01.442067 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.442042    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:46:01.442202 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:01.442165    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sll74" podUID="ce358ba2-d29f-4719-b826-c8726759568c"
Sep 16 18:46:01.760692 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.760657    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-222.ec2.internal" event="NodeReady"
Sep 16 18:46:01.761102 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.760782    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:46:01.832830 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.832784    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v6bsb"]
Sep 16 18:46:01.838353 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.838327    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zp57f"]
Sep 16 18:46:01.838495 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.838470    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v6bsb"
Sep 16 18:46:01.841150 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.841129    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:46:01.841312 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.841292    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dts4n\""
Sep 16 18:46:01.841442 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.841424    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:46:01.841537 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.841501    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:46:01.842224 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.842206    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:01.844438 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.844398    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:46:01.844622 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.844597    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v6bsb"]
Sep 16 18:46:01.844689 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.844624    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:46:01.844689 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.844597    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8559\""
Sep 16 18:46:01.849501 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.849429    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zp57f"]
Sep 16 18:46:01.926208 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.926170    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49314118-b6b2-47ce-b1c4-e5dd1140a71a-cert\") pod \"ingress-canary-v6bsb\" (UID: \"49314118-b6b2-47ce-b1c4-e5dd1140a71a\") " pod="openshift-ingress-canary/ingress-canary-v6bsb"
Sep 16 18:46:01.926355 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.926230    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjv8v\" (UniqueName: \"kubernetes.io/projected/49314118-b6b2-47ce-b1c4-e5dd1140a71a-kube-api-access-bjv8v\") pod \"ingress-canary-v6bsb\" (UID: \"49314118-b6b2-47ce-b1c4-e5dd1140a71a\") " pod="openshift-ingress-canary/ingress-canary-v6bsb"
Sep 16 18:46:01.926355 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.926285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-config-volume\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:01.926355 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.926326    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkcd5\" (UniqueName: \"kubernetes.io/projected/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-kube-api-access-mkcd5\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:01.926355 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.926347    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-tmp-dir\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:01.926498 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:01.926413    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:02.027330 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.027292    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:02.027499 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.027344    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49314118-b6b2-47ce-b1c4-e5dd1140a71a-cert\") pod \"ingress-canary-v6bsb\" (UID: \"49314118-b6b2-47ce-b1c4-e5dd1140a71a\") " pod="openshift-ingress-canary/ingress-canary-v6bsb"
Sep 16 18:46:02.027499 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.027379    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjv8v\" (UniqueName: \"kubernetes.io/projected/49314118-b6b2-47ce-b1c4-e5dd1140a71a-kube-api-access-bjv8v\") pod \"ingress-canary-v6bsb\" (UID: \"49314118-b6b2-47ce-b1c4-e5dd1140a71a\") " pod="openshift-ingress-canary/ingress-canary-v6bsb"
Sep 16 18:46:02.027499 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.027409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-config-volume\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:02.027499 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.027448    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcd5\" (UniqueName: \"kubernetes.io/projected/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-kube-api-access-mkcd5\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:02.027499 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:02.027454    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:02.027499 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.027473    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-tmp-dir\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:02.027818 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:02.027606    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls podName:b713bc4f-84ae-422e-8c29-0e37ac9f5f8d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:02.527580319 +0000 UTC m=+36.598677540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls") pod "dns-default-zp57f" (UID: "b713bc4f-84ae-422e-8c29-0e37ac9f5f8d") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:02.027919 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.027901    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-tmp-dir\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:02.028057 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.028041    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-config-volume\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:02.032307 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.032282    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49314118-b6b2-47ce-b1c4-e5dd1140a71a-cert\") pod \"ingress-canary-v6bsb\" (UID: \"49314118-b6b2-47ce-b1c4-e5dd1140a71a\") " pod="openshift-ingress-canary/ingress-canary-v6bsb"
Sep 16 18:46:02.038301 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.038276    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcd5\" (UniqueName: \"kubernetes.io/projected/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-kube-api-access-mkcd5\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:02.038658 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.038629    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjv8v\" (UniqueName: \"kubernetes.io/projected/49314118-b6b2-47ce-b1c4-e5dd1140a71a-kube-api-access-bjv8v\") pod \"ingress-canary-v6bsb\" (UID: \"49314118-b6b2-47ce-b1c4-e5dd1140a71a\") " pod="openshift-ingress-canary/ingress-canary-v6bsb"
Sep 16 18:46:02.151969 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.151930    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v6bsb"
Sep 16 18:46:02.442202 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.442121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:46:02.442202 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.442164    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:46:02.447322 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.446995    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:02.447322 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.447169    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:02.447322 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.447232    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtjvb\""
Sep 16 18:46:02.447587 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.447430    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:02.449085 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.449061    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqjhk\""
Sep 16 18:46:02.531860 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.531825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:02.532027 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:02.531968    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:02.532075 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:02.532036    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls podName:b713bc4f-84ae-422e-8c29-0e37ac9f5f8d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:03.532016732 +0000 UTC m=+37.603113948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls") pod "dns-default-zp57f" (UID: "b713bc4f-84ae-422e-8c29-0e37ac9f5f8d") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:02.919713 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:02.919543    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v6bsb"]
Sep 16 18:46:02.932374 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:46:02.932348    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49314118_b6b2_47ce_b1c4_e5dd1140a71a.slice/crio-ed5ad2901fcbdadebc2acaa77aa7eac526a094d6b7bf2f7e241277a1ad6cbd57 WatchSource:0}: Error finding container ed5ad2901fcbdadebc2acaa77aa7eac526a094d6b7bf2f7e241277a1ad6cbd57: Status 404 returned error can't find the container with id ed5ad2901fcbdadebc2acaa77aa7eac526a094d6b7bf2f7e241277a1ad6cbd57
Sep 16 18:46:03.441647 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:03.441566    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:46:03.444237 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:03.444211    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:03.541869 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:03.541830    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:03.542037 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:03.541987    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:03.542102 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:03.542070    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls podName:b713bc4f-84ae-422e-8c29-0e37ac9f5f8d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:05.542046926 +0000 UTC m=+39.613144148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls") pod "dns-default-zp57f" (UID: "b713bc4f-84ae-422e-8c29-0e37ac9f5f8d") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:03.779812 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:03.779775    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb01d6a2-e6d0-41b6-944a-7158baacf6a0" containerID="f3fbbc3dfcc207d7acdb92c80367f6f9a6471b96c1fbcddc2237d939887b85a5" exitCode=0
Sep 16 18:46:03.779986 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:03.779859    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdj6m" event={"ID":"fb01d6a2-e6d0-41b6-944a-7158baacf6a0","Type":"ContainerDied","Data":"f3fbbc3dfcc207d7acdb92c80367f6f9a6471b96c1fbcddc2237d939887b85a5"}
Sep 16 18:46:03.781155 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:03.781135    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v6bsb" event={"ID":"49314118-b6b2-47ce-b1c4-e5dd1140a71a","Type":"ContainerStarted","Data":"ed5ad2901fcbdadebc2acaa77aa7eac526a094d6b7bf2f7e241277a1ad6cbd57"}
Sep 16 18:46:04.787160 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:04.787115    2572 generic.go:356] "Generic (PLEG): container finished" podID="fb01d6a2-e6d0-41b6-944a-7158baacf6a0" containerID="9892bed1f802d6c46574e220feb56d0ac95b8eabe3fdc533d448b50f4cf06846" exitCode=0
Sep 16 18:46:04.787783 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:04.787186    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdj6m" event={"ID":"fb01d6a2-e6d0-41b6-944a-7158baacf6a0","Type":"ContainerDied","Data":"9892bed1f802d6c46574e220feb56d0ac95b8eabe3fdc533d448b50f4cf06846"}
Sep 16 18:46:05.559481 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:05.559395    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:05.559655 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:05.559579    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:05.559655 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:05.559647    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls podName:b713bc4f-84ae-422e-8c29-0e37ac9f5f8d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:09.559632362 +0000 UTC m=+43.630729564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls") pod "dns-default-zp57f" (UID: "b713bc4f-84ae-422e-8c29-0e37ac9f5f8d") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:05.792670 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:05.792631    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fdj6m" event={"ID":"fb01d6a2-e6d0-41b6-944a-7158baacf6a0","Type":"ContainerStarted","Data":"0ae61fe4707cc9ce3d00f814f6c1c5ed3c864b4d02898481aae4d98e2ba24397"}
Sep 16 18:46:05.794136 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:05.794103    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v6bsb" event={"ID":"49314118-b6b2-47ce-b1c4-e5dd1140a71a","Type":"ContainerStarted","Data":"e231edd9c1955b6154e5828d92700b6d9cd30dcb4902aeb7485bee753855ce3d"}
Sep 16 18:46:05.815558 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:05.815443    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fdj6m" podStartSLOduration=8.496310808 podStartE2EDuration="38.815425394s" totalImagesPullingTime="30.319114586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:05.814542972 +0000 UTC m=+39.885640190" watchObservedRunningTime="2026-09-16 18:46:05.815425394 +0000 UTC m=+39.886522618"
Sep 16 18:46:05.832012 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:05.831962    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v6bsb" podStartSLOduration=2.502747184 podStartE2EDuration="4.831946701s" totalImagesPullingTime="2.329199517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:05.831190174 +0000 UTC m=+39.902287400" watchObservedRunningTime="2026-09-16 18:46:05.831946701 +0000 UTC m=+39.903043924"
Sep 16 18:46:09.588600 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:09.588346    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:09.589017 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:09.588500    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:09.589017 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:09.588720    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls podName:b713bc4f-84ae-422e-8c29-0e37ac9f5f8d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.588706184 +0000 UTC m=+51.659803400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls") pod "dns-default-zp57f" (UID: "b713bc4f-84ae-422e-8c29-0e37ac9f5f8d") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:17.646447 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:17.646408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:17.646884 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:17.646574    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:17.646884 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:17.646650    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls podName:b713bc4f-84ae-422e-8c29-0e37ac9f5f8d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:33.64663443 +0000 UTC m=+67.717731632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls") pod "dns-default-zp57f" (UID: "b713bc4f-84ae-422e-8c29-0e37ac9f5f8d") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:27.770141 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:27.770107    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wc29d"
Sep 16 18:46:32.147903 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.147853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:46:32.148367 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.147946    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:46:32.150404 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.150383    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:32.150736 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.150720    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:32.158705 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:32.158687    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:46:32.158799 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:32.158753    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs podName:2c4cd60c-ec4c-4284-b45e-bb39c7786cf8 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:36.158734932 +0000 UTC m=+130.229832148 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs") pod "network-metrics-daemon-856sw" (UID: "2c4cd60c-ec4c-4284-b45e-bb39c7786cf8") : secret "metrics-daemon-secret" not found
Sep 16 18:46:32.162087 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.162064    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce358ba2-d29f-4719-b826-c8726759568c-original-pull-secret\") pod \"global-pull-secret-syncer-sll74\" (UID: \"ce358ba2-d29f-4719-b826-c8726759568c\") " pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:46:32.252136 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.252099    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sll74"
Sep 16 18:46:32.349471 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.349431    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:46:32.352039 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.352021    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:32.362108 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.362090    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:32.373428 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.373402    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m88pl\" (UniqueName: \"kubernetes.io/projected/0c903d9e-cd3d-4818-be5d-299cfd855b2e-kube-api-access-m88pl\") pod \"network-check-target-sz5vt\" (UID: \"0c903d9e-cd3d-4818-be5d-299cfd855b2e\") " pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:46:32.463173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.463142    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqjhk\""
Sep 16 18:46:32.471375 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.471356    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:46:32.523821 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.523787    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sll74"]
Sep 16 18:46:32.754681 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.754611    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sz5vt"]
Sep 16 18:46:32.757582 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:46:32.757546    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c903d9e_cd3d_4818_be5d_299cfd855b2e.slice/crio-f25c4c5d0c9aefc19c33ba7f611de752cfa5443515433e1b77c9648362c6896c WatchSource:0}: Error finding container f25c4c5d0c9aefc19c33ba7f611de752cfa5443515433e1b77c9648362c6896c: Status 404 returned error can't find the container with id f25c4c5d0c9aefc19c33ba7f611de752cfa5443515433e1b77c9648362c6896c
Sep 16 18:46:32.857126 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.857095    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sz5vt" event={"ID":"0c903d9e-cd3d-4818-be5d-299cfd855b2e","Type":"ContainerStarted","Data":"f25c4c5d0c9aefc19c33ba7f611de752cfa5443515433e1b77c9648362c6896c"}
Sep 16 18:46:32.858192 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:32.858166    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sll74" event={"ID":"ce358ba2-d29f-4719-b826-c8726759568c","Type":"ContainerStarted","Data":"e7c16bbeea63c1aadad726a8a80098cadcdf949e27dccb80349f99ece97c58e7"}
Sep 16 18:46:33.658797 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:33.658753    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:46:33.659234 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:33.658900    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:33.659234 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:33.658964    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls podName:b713bc4f-84ae-422e-8c29-0e37ac9f5f8d nodeName:}" failed. No retries permitted until 2026-09-16 18:47:05.658947304 +0000 UTC m=+99.730044525 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls") pod "dns-default-zp57f" (UID: "b713bc4f-84ae-422e-8c29-0e37ac9f5f8d") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:37.871125 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:37.871079    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sz5vt" event={"ID":"0c903d9e-cd3d-4818-be5d-299cfd855b2e","Type":"ContainerStarted","Data":"251963a9ed38afe23a22d2feb79689337af42d0a451b3b6dac1812c42cd7198a"}
Sep 16 18:46:37.871640 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:37.871208    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:46:37.872718 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:37.872694    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sll74" event={"ID":"ce358ba2-d29f-4719-b826-c8726759568c","Type":"ContainerStarted","Data":"77808079c03a1c5683b097b342c79d14e1e2d803600a7177b033a1b147115020"}
Sep 16 18:46:37.889012 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:37.888963    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sz5vt" podStartSLOduration=66.267613269 podStartE2EDuration="1m10.888945406s" totalImagesPullingTime="4.621332137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:37.888653092 +0000 UTC m=+71.959750318" watchObservedRunningTime="2026-09-16 18:46:37.888945406 +0000 UTC m=+71.960042631"
Sep 16 18:46:37.905688 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:37.905648    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sll74" podStartSLOduration=66.055714202 podStartE2EDuration="1m10.905633117s" totalImagesPullingTime="4.849918915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:37.905286735 +0000 UTC m=+71.976383960" watchObservedRunningTime="2026-09-16 18:46:37.905633117 +0000 UTC m=+71.976730335"
Sep 16 18:46:57.734000 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.733735    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"]
Sep 16 18:46:57.737926 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.737902    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.741068 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.741049    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:46:57.741363 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.741343    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:46:57.741675 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.741661    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:57.741863 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.741851    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tv2sr\""
Sep 16 18:46:57.742467 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.742452    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:57.747531 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.747491    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"]
Sep 16 18:46:57.821297 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.821245    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrzx\" (UniqueName: \"kubernetes.io/projected/3c09636a-36dd-41e9-a35b-41831a48fc4a-kube-api-access-4jrzx\") pod \"service-ca-operator-cf7c776b7-2xpx5\" (UID: \"3c09636a-36dd-41e9-a35b-41831a48fc4a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.821297 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.821299    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c09636a-36dd-41e9-a35b-41831a48fc4a-config\") pod \"service-ca-operator-cf7c776b7-2xpx5\" (UID: \"3c09636a-36dd-41e9-a35b-41831a48fc4a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.821577 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.821374    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c09636a-36dd-41e9-a35b-41831a48fc4a-serving-cert\") pod \"service-ca-operator-cf7c776b7-2xpx5\" (UID: \"3c09636a-36dd-41e9-a35b-41831a48fc4a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.846479 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.846448    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f9dc5957d-6j7sr"]
Sep 16 18:46:57.849230 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.849214    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:57.851553 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.851509    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:46:57.851644 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.851509    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:46:57.851644 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.851570    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:46:57.852346 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.852326    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:46:57.852407 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.852354    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:46:57.852407 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.852396    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:46:57.852538 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.852402    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pg2hn\""
Sep 16 18:46:57.863963 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.863942    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f9dc5957d-6j7sr"]
Sep 16 18:46:57.922034 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.922009    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5n7h\" (UniqueName: \"kubernetes.io/projected/783e7bea-f22c-4f77-9958-facda1f07bd9-kube-api-access-w5n7h\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:57.922228 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.922043    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:57.922228 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.922066    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c09636a-36dd-41e9-a35b-41831a48fc4a-serving-cert\") pod \"service-ca-operator-cf7c776b7-2xpx5\" (UID: \"3c09636a-36dd-41e9-a35b-41831a48fc4a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.922228 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.922096    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:57.922228 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.922171    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrzx\" (UniqueName: \"kubernetes.io/projected/3c09636a-36dd-41e9-a35b-41831a48fc4a-kube-api-access-4jrzx\") pod \"service-ca-operator-cf7c776b7-2xpx5\" (UID: \"3c09636a-36dd-41e9-a35b-41831a48fc4a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.922228 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.922212    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-default-certificate\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:57.922465 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.922238    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-stats-auth\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:57.922465 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.922327    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c09636a-36dd-41e9-a35b-41831a48fc4a-config\") pod \"service-ca-operator-cf7c776b7-2xpx5\" (UID: \"3c09636a-36dd-41e9-a35b-41831a48fc4a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.922827 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.922806    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c09636a-36dd-41e9-a35b-41831a48fc4a-config\") pod \"service-ca-operator-cf7c776b7-2xpx5\" (UID: \"3c09636a-36dd-41e9-a35b-41831a48fc4a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.924494 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.924475    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c09636a-36dd-41e9-a35b-41831a48fc4a-serving-cert\") pod \"service-ca-operator-cf7c776b7-2xpx5\" (UID: \"3c09636a-36dd-41e9-a35b-41831a48fc4a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.937822 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.937793    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrzx\" (UniqueName: \"kubernetes.io/projected/3c09636a-36dd-41e9-a35b-41831a48fc4a-kube-api-access-4jrzx\") pod \"service-ca-operator-cf7c776b7-2xpx5\" (UID: \"3c09636a-36dd-41e9-a35b-41831a48fc4a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:57.939837 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.939816    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"]
Sep 16 18:46:57.942901 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.942800    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:57.943813 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.943791    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"]
Sep 16 18:46:57.946065 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.946047    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:46:57.946065 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.946059    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:57.946749 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.946717    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:57.946863 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.946821    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gd7c2\""
Sep 16 18:46:57.947040 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.947022    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:46:57.947928 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.947903    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:57.948655 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.948627    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:46:57.950425 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.950382    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:46:57.950806 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.950786    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fjkbf\""
Sep 16 18:46:57.950883 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.950808    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:46:57.950883 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.950730    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:46:57.955150 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.955127    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"]
Sep 16 18:46:57.962281 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:57.962259    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"]
Sep 16 18:46:58.023223 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023140    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.023223 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-default-certificate\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.023223 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-stats-auth\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.023461 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023228    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab856818-b5aa-4b90-b7ac-fc9743acf780-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:58.023461 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023248    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:58.023461 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n825t\" (UID: \"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.023461 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.023308    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:58.023461 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023338    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5n7h\" (UniqueName: \"kubernetes.io/projected/783e7bea-f22c-4f77-9958-facda1f07bd9-kube-api-access-w5n7h\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.023461 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023381    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb7mp\" (UniqueName: \"kubernetes.io/projected/ab856818-b5aa-4b90-b7ac-fc9743acf780-kube-api-access-zb7mp\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:58.023461 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.023395    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:58.523376685 +0000 UTC m=+92.594473891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : secret "router-metrics-certs-default" not found
Sep 16 18:46:58.023461 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023441    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wssqj\" (UniqueName: \"kubernetes.io/projected/71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff-kube-api-access-wssqj\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n825t\" (UID: \"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.023783 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.023783 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.023532    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n825t\" (UID: \"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.023783 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.023624    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:58.523606396 +0000 UTC m=+92.594703597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:58.025861 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.025840    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-default-certificate\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.025973 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.025884    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-stats-auth\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.032060 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.032029    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5n7h\" (UniqueName: \"kubernetes.io/projected/783e7bea-f22c-4f77-9958-facda1f07bd9-kube-api-access-w5n7h\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.046654 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.046627    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"
Sep 16 18:46:58.123991 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.123960    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wssqj\" (UniqueName: \"kubernetes.io/projected/71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff-kube-api-access-wssqj\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n825t\" (UID: \"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.124162 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.124011    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n825t\" (UID: \"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.124162 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.124070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab856818-b5aa-4b90-b7ac-fc9743acf780-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:58.124162 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.124086    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:58.124162 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.124105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n825t\" (UID: \"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.124162 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.124132    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb7mp\" (UniqueName: \"kubernetes.io/projected/ab856818-b5aa-4b90-b7ac-fc9743acf780-kube-api-access-zb7mp\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:58.124419 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.124196    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:58.124419 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.124268    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls podName:ab856818-b5aa-4b90-b7ac-fc9743acf780 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:58.624247238 +0000 UTC m=+92.695344443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fnzwg" (UID: "ab856818-b5aa-4b90-b7ac-fc9743acf780") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:58.124748 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.124728    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n825t\" (UID: \"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.124831 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.124804    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab856818-b5aa-4b90-b7ac-fc9743acf780-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:58.126362 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.126342    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n825t\" (UID: \"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.132636 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.132615    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wssqj\" (UniqueName: \"kubernetes.io/projected/71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff-kube-api-access-wssqj\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n825t\" (UID: \"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.134984 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.134965    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb7mp\" (UniqueName: \"kubernetes.io/projected/ab856818-b5aa-4b90-b7ac-fc9743acf780-kube-api-access-zb7mp\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:58.256624 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.256590    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"
Sep 16 18:46:58.320491 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.320464    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5"]
Sep 16 18:46:58.324194 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:46:58.324166    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c09636a_36dd_41e9_a35b_41831a48fc4a.slice/crio-0f117464c7b5dd85d2e85c163fe02a25106587d5c952244e908e411a39968133 WatchSource:0}: Error finding container 0f117464c7b5dd85d2e85c163fe02a25106587d5c952244e908e411a39968133: Status 404 returned error can't find the container with id 0f117464c7b5dd85d2e85c163fe02a25106587d5c952244e908e411a39968133
Sep 16 18:46:58.526662 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.526627    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.526866 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.526692    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:58.526866 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.526850    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:59.526828796 +0000 UTC m=+93.597926011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:58.527062 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.527042    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:58.527119 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.527100    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:59.52708502 +0000 UTC m=+93.598182242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : secret "router-metrics-certs-default" not found
Sep 16 18:46:58.527795 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.527773    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t"]
Sep 16 18:46:58.530701 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:46:58.530673    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b869ae_cd5b_4bc0_89b7_6164c7dbb0ff.slice/crio-a8b55fe7ba559a191d97d7fa7ee7fd2d625fe9f1aafc968e9494534482f705fc WatchSource:0}: Error finding container a8b55fe7ba559a191d97d7fa7ee7fd2d625fe9f1aafc968e9494534482f705fc: Status 404 returned error can't find the container with id a8b55fe7ba559a191d97d7fa7ee7fd2d625fe9f1aafc968e9494534482f705fc
Sep 16 18:46:58.627867 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.627776    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:58.628008 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.627932    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:58.628008 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:58.628005    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls podName:ab856818-b5aa-4b90-b7ac-fc9743acf780 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:59.6279879 +0000 UTC m=+93.699085101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fnzwg" (UID: "ab856818-b5aa-4b90-b7ac-fc9743acf780") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:58.927424 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.927347    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5" event={"ID":"3c09636a-36dd-41e9-a35b-41831a48fc4a","Type":"ContainerStarted","Data":"0f117464c7b5dd85d2e85c163fe02a25106587d5c952244e908e411a39968133"}
Sep 16 18:46:58.928806 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:58.928775    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t" event={"ID":"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff","Type":"ContainerStarted","Data":"a8b55fe7ba559a191d97d7fa7ee7fd2d625fe9f1aafc968e9494534482f705fc"}
Sep 16 18:46:59.536320 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:59.535980    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:59.536320 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:59.536060    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:46:59.536320 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:59.536176    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:01.536149377 +0000 UTC m=+95.607246598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:59.536320 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:59.536238    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:59.536320 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:59.536294    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:01.536276627 +0000 UTC m=+95.607373840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : secret "router-metrics-certs-default" not found
Sep 16 18:46:59.636812 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:46:59.636778    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:46:59.636964 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:59.636914    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:46:59.637028 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:46:59.636977    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls podName:ab856818-b5aa-4b90-b7ac-fc9743acf780 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:01.636961944 +0000 UTC m=+95.708059145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fnzwg" (UID: "ab856818-b5aa-4b90-b7ac-fc9743acf780") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:47:00.935388 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:00.935354    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5" event={"ID":"3c09636a-36dd-41e9-a35b-41831a48fc4a","Type":"ContainerStarted","Data":"602748e9543276b8371c5bff942db7437aa5d19516e79f97b7738106fe70b633"}
Sep 16 18:47:00.938399 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:00.938172    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t" event={"ID":"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff","Type":"ContainerStarted","Data":"4d34aa761c58c6829bedf5806795f80047b2058b5b0efd28cff68dafe667cf63"}
Sep 16 18:47:00.952949 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:00.952790    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5" podStartSLOduration=1.432103747 podStartE2EDuration="3.952771737s" totalImagesPullingTime="2.52066799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:00.95237199 +0000 UTC m=+95.023469215" watchObservedRunningTime="2026-09-16 18:47:00.952771737 +0000 UTC m=+95.023868962"
Sep 16 18:47:00.973998 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:00.973944    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t" podStartSLOduration=1.655329345 podStartE2EDuration="3.973926059s" totalImagesPullingTime="2.318596714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:00.971888762 +0000 UTC m=+95.042985987" watchObservedRunningTime="2026-09-16 18:47:00.973926059 +0000 UTC m=+95.045023284"
Sep 16 18:47:01.552484 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:01.552448    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:01.552662 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:01.552494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:01.552662 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:01.552611    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:47:01.552662 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:01.552625    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:05.55260779 +0000 UTC m=+99.623704995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:47:01.552662 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:01.552652    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:05.552640311 +0000 UTC m=+99.623737513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : secret "router-metrics-certs-default" not found
Sep 16 18:47:01.653097 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:01.653039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:47:01.653297 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:01.653189    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:47:01.653297 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:01.653258    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls podName:ab856818-b5aa-4b90-b7ac-fc9743acf780 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:05.6532432 +0000 UTC m=+99.724340401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fnzwg" (UID: "ab856818-b5aa-4b90-b7ac-fc9743acf780") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:47:02.164461 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.164423    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k"]
Sep 16 18:47:02.167755 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.167732    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k"
Sep 16 18:47:02.170013 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.169993    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:47:02.170113 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.170042    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:47:02.170714 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.170692    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-txlfx\""
Sep 16 18:47:02.185905 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.185884    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k"]
Sep 16 18:47:02.258484 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.258458    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xtd8\" (UniqueName: \"kubernetes.io/projected/b085dfcf-c262-451c-8e7e-1bf92f7164f6-kube-api-access-7xtd8\") pod \"migrator-8b4d697c5-s7h8k\" (UID: \"b085dfcf-c262-451c-8e7e-1bf92f7164f6\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k"
Sep 16 18:47:02.359217 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.359180    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xtd8\" (UniqueName: \"kubernetes.io/projected/b085dfcf-c262-451c-8e7e-1bf92f7164f6-kube-api-access-7xtd8\") pod \"migrator-8b4d697c5-s7h8k\" (UID: \"b085dfcf-c262-451c-8e7e-1bf92f7164f6\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k"
Sep 16 18:47:02.368426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.368400    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xtd8\" (UniqueName: \"kubernetes.io/projected/b085dfcf-c262-451c-8e7e-1bf92f7164f6-kube-api-access-7xtd8\") pod \"migrator-8b4d697c5-s7h8k\" (UID: \"b085dfcf-c262-451c-8e7e-1bf92f7164f6\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k"
Sep 16 18:47:02.477426 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.477395    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k"
Sep 16 18:47:02.743105 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.743023    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l97v6"]
Sep 16 18:47:02.746477 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.746453    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.749379 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.749362    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:47:02.749673 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.749654    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-46pc5\""
Sep 16 18:47:02.749935 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.749921    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:47:02.750691 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.750673    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:47:02.752674 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.752492    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:47:02.752674 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.752638    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k"]
Sep 16 18:47:02.757303 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:02.757283    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb085dfcf_c262_451c_8e7e_1bf92f7164f6.slice/crio-b2b154e7cc4a162327a9a01928379cff9ffc33661a01564e3ccd6cdc3c30abfa WatchSource:0}: Error finding container b2b154e7cc4a162327a9a01928379cff9ffc33661a01564e3ccd6cdc3c30abfa: Status 404 returned error can't find the container with id b2b154e7cc4a162327a9a01928379cff9ffc33661a01564e3ccd6cdc3c30abfa
Sep 16 18:47:02.760397 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.760334    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l97v6"]
Sep 16 18:47:02.863325 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.863288    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7df4f878-1807-43fd-acdd-3a6cff6bc930-crio-socket\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.863325 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.863327    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7jhh\" (UniqueName: \"kubernetes.io/projected/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-api-access-q7jhh\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.863593 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.863404    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.863593 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.863446    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.863593 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.863471    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7df4f878-1807-43fd-acdd-3a6cff6bc930-data-volume\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.944631 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.944595    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k" event={"ID":"b085dfcf-c262-451c-8e7e-1bf92f7164f6","Type":"ContainerStarted","Data":"b2b154e7cc4a162327a9a01928379cff9ffc33661a01564e3ccd6cdc3c30abfa"}
Sep 16 18:47:02.964652 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.964622    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.964788 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.964664    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.964788 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.964695    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7df4f878-1807-43fd-acdd-3a6cff6bc930-data-volume\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.964788 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.964735    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7df4f878-1807-43fd-acdd-3a6cff6bc930-crio-socket\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.964788 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:02.964744    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:02.964788 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.964768    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7jhh\" (UniqueName: \"kubernetes.io/projected/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-api-access-q7jhh\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.964940 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:02.964793    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:02.964940 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:02.964805    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:03.464783599 +0000 UTC m=+97.535880805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:02.964940 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:02.964846    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:03.4648304 +0000 UTC m=+97.535927620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:02.964940 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.964846    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7df4f878-1807-43fd-acdd-3a6cff6bc930-crio-socket\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.965121 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.965106    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7df4f878-1807-43fd-acdd-3a6cff6bc930-data-volume\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:02.977289 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:02.977262    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7jhh\" (UniqueName: \"kubernetes.io/projected/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-api-access-q7jhh\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:03.468648 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:03.468599    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:03.469022 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:03.468676    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:03.469022 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:03.468722    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:03.469022 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:03.468791    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.468774728 +0000 UTC m=+98.539871930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:03.469022 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:03.468823    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:03.469022 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:03.468891    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.468877334 +0000 UTC m=+98.539974541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:04.477266 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:04.477224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:04.477754 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:04.477323    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:04.477754 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:04.477393    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:04.477754 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:04.477399    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:04.477754 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:04.477446    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:06.477432133 +0000 UTC m=+100.548529334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:04.477754 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:04.477471    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:06.477451985 +0000 UTC m=+100.548549187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:04.952233 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:04.952207    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k" event={"ID":"b085dfcf-c262-451c-8e7e-1bf92f7164f6","Type":"ContainerStarted","Data":"0698cf04ef4a147f69da966565a11f63f7dfca78016a691da4dc6c198d471581"}
Sep 16 18:47:04.952385 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:04.952263    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k" event={"ID":"b085dfcf-c262-451c-8e7e-1bf92f7164f6","Type":"ContainerStarted","Data":"3149fd86bdf4ec86e308f729259403b249f968c835cfa4de100d32038822bda8"}
Sep 16 18:47:04.970361 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:04.970313    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-s7h8k" podStartSLOduration=1.623121162 podStartE2EDuration="2.970298s" totalImagesPullingTime="1.347176838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:04.969099081 +0000 UTC m=+99.040196305" watchObservedRunningTime="2026-09-16 18:47:04.970298 +0000 UTC m=+99.041395224"
Sep 16 18:47:05.221419 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:05.221346    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fjbx_b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368/dns-node-resolver/0.log"
Sep 16 18:47:05.587183 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:05.587091    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:05.587183 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:05.587141    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:05.587595 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:05.587257    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:47:05.587595 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:05.587271    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:13.587252315 +0000 UTC m=+107.658349530 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:47:05.587595 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:05.587297    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:13.587286973 +0000 UTC m=+107.658384174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : secret "router-metrics-certs-default" not found
Sep 16 18:47:05.687676 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:05.687646    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:47:05.687822 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:05.687728    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:47:05.687822 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:05.687806    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:47:05.687888 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:05.687864    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls podName:ab856818-b5aa-4b90-b7ac-fc9743acf780 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:13.687849917 +0000 UTC m=+107.758947119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fnzwg" (UID: "ab856818-b5aa-4b90-b7ac-fc9743acf780") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:47:05.687953 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:05.687806    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:47:05.687953 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:05.687928    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls podName:b713bc4f-84ae-422e-8c29-0e37ac9f5f8d nodeName:}" failed. No retries permitted until 2026-09-16 18:48:09.687914599 +0000 UTC m=+163.759011818 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls") pod "dns-default-zp57f" (UID: "b713bc4f-84ae-422e-8c29-0e37ac9f5f8d") : secret "dns-default-metrics-tls" not found
Sep 16 18:47:06.424928 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:06.424902    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xtfbc_67557c8c-a566-40ca-b64c-8760f4fbbbe7/node-ca/0.log"
Sep 16 18:47:06.494964 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:06.494926    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:06.495155 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:06.494977    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:06.495155 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:06.495059    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:06.495155 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:06.495086    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:06.495155 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:06.495133    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.495112415 +0000 UTC m=+104.566209635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:06.495155 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:06.495153    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.495144045 +0000 UTC m=+104.566241246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:07.425055 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:07.425022    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v6bsb_49314118-b6b2-47ce-b1c4-e5dd1140a71a/serve-healthcheck-canary/0.log"
Sep 16 18:47:07.622681 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:07.622653    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-s7h8k_b085dfcf-c262-451c-8e7e-1bf92f7164f6/migrator/0.log"
Sep 16 18:47:07.823689 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:07.823656    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-s7h8k_b085dfcf-c262-451c-8e7e-1bf92f7164f6/graceful-termination/0.log"
Sep 16 18:47:08.023817 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:08.023791    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-n825t_71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff/kube-storage-version-migrator-operator/0.log"
Sep 16 18:47:08.877985 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:08.877956    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sz5vt"
Sep 16 18:47:10.530026 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:10.529996    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:10.530414 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:10.530106    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:10.530414 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:10.530166    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:10.530414 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:10.530189    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:47:10.530414 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:10.530237    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:18.530220532 +0000 UTC m=+112.601317734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : configmap "kube-rbac-proxy" not found
Sep 16 18:47:10.530414 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:10.530253    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:18.53024552 +0000 UTC m=+112.601342721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:13.657322 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:13.657276    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:13.657322 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:13.657327    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:13.657753 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:13.657443    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:47:13.657753 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:13.657447    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:29.657428775 +0000 UTC m=+123.728525976 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:47:13.657753 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:13.657505    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs podName:783e7bea-f22c-4f77-9958-facda1f07bd9 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:29.657486911 +0000 UTC m=+123.728584119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs") pod "router-default-7f9dc5957d-6j7sr" (UID: "783e7bea-f22c-4f77-9958-facda1f07bd9") : secret "router-metrics-certs-default" not found
Sep 16 18:47:13.758601 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:13.758555    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:47:13.758778 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:13.758689    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:47:13.758778 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:13.758752    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls podName:ab856818-b5aa-4b90-b7ac-fc9743acf780 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:29.75873687 +0000 UTC m=+123.829834072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-fnzwg" (UID: "ab856818-b5aa-4b90-b7ac-fc9743acf780") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:47:18.595622 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:18.595578    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:18.596050 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:18.595640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:18.596050 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:18.595741    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:18.596050 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:18.595822    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls podName:7df4f878-1807-43fd-acdd-3a6cff6bc930 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:34.595806579 +0000 UTC m=+128.666903780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l97v6" (UID: "7df4f878-1807-43fd-acdd-3a6cff6bc930") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:47:18.596160 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:18.596106    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7df4f878-1807-43fd-acdd-3a6cff6bc930-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:29.267473 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.267437    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-95c57"]
Sep 16 18:47:29.270481 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.270460    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-95c57"
Sep 16 18:47:29.282074 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.282049    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:47:29.282753 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.282738    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:47:29.283097 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.283083    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hxkjn\""
Sep 16 18:47:29.303328 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.303302    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-95c57"]
Sep 16 18:47:29.334797 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.334758    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85967c5989-mn8js"]
Sep 16 18:47:29.338129 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.338097    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.343190 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.343158    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:47:29.343467 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.343446    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:47:29.345001 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.344977    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-htp7b\""
Sep 16 18:47:29.351548 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.351502    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:47:29.357675 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.357649    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:47:29.363933 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.363901    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85967c5989-mn8js"]
Sep 16 18:47:29.378145 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.378100    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6pd7\" (UniqueName: \"kubernetes.io/projected/d1108c81-4a31-4a0f-8856-419c30c68edb-kube-api-access-j6pd7\") pod \"downloads-76f99584f4-95c57\" (UID: \"d1108c81-4a31-4a0f-8856-419c30c68edb\") " pod="openshift-console/downloads-76f99584f4-95c57"
Sep 16 18:47:29.478991 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.478944    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d793038-dc8f-45d8-a3b9-af84ae589c23-registry-certificates\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.478991 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.478999    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d793038-dc8f-45d8-a3b9-af84ae589c23-image-registry-private-configuration\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.479229 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.479052    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d793038-dc8f-45d8-a3b9-af84ae589c23-bound-sa-token\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.479229 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.479112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d793038-dc8f-45d8-a3b9-af84ae589c23-registry-tls\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.479229 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.479185    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d793038-dc8f-45d8-a3b9-af84ae589c23-trusted-ca\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.479229 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.479225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmzrx\" (UniqueName: \"kubernetes.io/projected/5d793038-dc8f-45d8-a3b9-af84ae589c23-kube-api-access-hmzrx\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.479380 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.479245    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d793038-dc8f-45d8-a3b9-af84ae589c23-installation-pull-secrets\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.479380 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.479279    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6pd7\" (UniqueName: \"kubernetes.io/projected/d1108c81-4a31-4a0f-8856-419c30c68edb-kube-api-access-j6pd7\") pod \"downloads-76f99584f4-95c57\" (UID: \"d1108c81-4a31-4a0f-8856-419c30c68edb\") " pod="openshift-console/downloads-76f99584f4-95c57"
Sep 16 18:47:29.479380 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.479306    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d793038-dc8f-45d8-a3b9-af84ae589c23-ca-trust-extracted\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.494302 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.494266    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6pd7\" (UniqueName: \"kubernetes.io/projected/d1108c81-4a31-4a0f-8856-419c30c68edb-kube-api-access-j6pd7\") pod \"downloads-76f99584f4-95c57\" (UID: \"d1108c81-4a31-4a0f-8856-419c30c68edb\") " pod="openshift-console/downloads-76f99584f4-95c57"
Sep 16 18:47:29.578938 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.578852    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-95c57"
Sep 16 18:47:29.579647 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.579627    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d793038-dc8f-45d8-a3b9-af84ae589c23-image-registry-private-configuration\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.579719 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.579664    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d793038-dc8f-45d8-a3b9-af84ae589c23-bound-sa-token\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.579719 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.579694    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d793038-dc8f-45d8-a3b9-af84ae589c23-registry-tls\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.579799 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.579725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d793038-dc8f-45d8-a3b9-af84ae589c23-trusted-ca\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.579799 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.579749    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmzrx\" (UniqueName: \"kubernetes.io/projected/5d793038-dc8f-45d8-a3b9-af84ae589c23-kube-api-access-hmzrx\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.579799 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.579766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d793038-dc8f-45d8-a3b9-af84ae589c23-installation-pull-secrets\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.579799 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.579796    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d793038-dc8f-45d8-a3b9-af84ae589c23-ca-trust-extracted\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.579987 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.579854    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d793038-dc8f-45d8-a3b9-af84ae589c23-registry-certificates\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.580325 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.580280    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d793038-dc8f-45d8-a3b9-af84ae589c23-ca-trust-extracted\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.580646 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.580623    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d793038-dc8f-45d8-a3b9-af84ae589c23-registry-certificates\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.580735 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.580673    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d793038-dc8f-45d8-a3b9-af84ae589c23-trusted-ca\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.582309 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.582291    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d793038-dc8f-45d8-a3b9-af84ae589c23-image-registry-private-configuration\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.582832 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.582810    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d793038-dc8f-45d8-a3b9-af84ae589c23-registry-tls\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.582920 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.582815    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d793038-dc8f-45d8-a3b9-af84ae589c23-installation-pull-secrets\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.590218 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.590196    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d793038-dc8f-45d8-a3b9-af84ae589c23-bound-sa-token\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.591475 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.591456    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmzrx\" (UniqueName: \"kubernetes.io/projected/5d793038-dc8f-45d8-a3b9-af84ae589c23-kube-api-access-hmzrx\") pod \"image-registry-85967c5989-mn8js\" (UID: \"5d793038-dc8f-45d8-a3b9-af84ae589c23\") " pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.656268 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.656238    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:29.680570 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.680536    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:29.680695 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.680603    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:29.681107 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.681078    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/783e7bea-f22c-4f77-9958-facda1f07bd9-service-ca-bundle\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:29.683103 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.683076    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/783e7bea-f22c-4f77-9958-facda1f07bd9-metrics-certs\") pod \"router-default-7f9dc5957d-6j7sr\" (UID: \"783e7bea-f22c-4f77-9958-facda1f07bd9\") " pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:29.781123 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.781085    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:47:29.783633 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.783605    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab856818-b5aa-4b90-b7ac-fc9743acf780-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-fnzwg\" (UID: \"ab856818-b5aa-4b90-b7ac-fc9743acf780\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:47:29.861203 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.861134    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-95c57"]
Sep 16 18:47:29.863500 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:29.863475    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1108c81_4a31_4a0f_8856_419c30c68edb.slice/crio-d767cc49f46ae828d634e027c9b305dd570f123ec6c3161d9ec0e391a77144bc WatchSource:0}: Error finding container d767cc49f46ae828d634e027c9b305dd570f123ec6c3161d9ec0e391a77144bc: Status 404 returned error can't find the container with id d767cc49f46ae828d634e027c9b305dd570f123ec6c3161d9ec0e391a77144bc
Sep 16 18:47:29.937233 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.937196    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85967c5989-mn8js"]
Sep 16 18:47:29.940018 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:29.939978    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d793038_dc8f_45d8_a3b9_af84ae589c23.slice/crio-c8461ef7e10bad07021e7eff59fc8c6fd075fe6960ef97d96b5f9ae853f9c224 WatchSource:0}: Error finding container c8461ef7e10bad07021e7eff59fc8c6fd075fe6960ef97d96b5f9ae853f9c224: Status 404 returned error can't find the container with id c8461ef7e10bad07021e7eff59fc8c6fd075fe6960ef97d96b5f9ae853f9c224
Sep 16 18:47:29.961211 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.961192    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pg2hn\""
Sep 16 18:47:29.968874 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:29.968858    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:30.033159 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:30.033132    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-95c57" event={"ID":"d1108c81-4a31-4a0f-8856-419c30c68edb","Type":"ContainerStarted","Data":"d767cc49f46ae828d634e027c9b305dd570f123ec6c3161d9ec0e391a77144bc"}
Sep 16 18:47:30.034970 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:30.034939    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85967c5989-mn8js" event={"ID":"5d793038-dc8f-45d8-a3b9-af84ae589c23","Type":"ContainerStarted","Data":"1d9214b31d2a315889b46c023ea71e5e8572124b900336d25e5c63ab8c05e2d7"}
Sep 16 18:47:30.035105 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:30.035001    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85967c5989-mn8js" event={"ID":"5d793038-dc8f-45d8-a3b9-af84ae589c23","Type":"ContainerStarted","Data":"c8461ef7e10bad07021e7eff59fc8c6fd075fe6960ef97d96b5f9ae853f9c224"}
Sep 16 18:47:30.035105 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:30.035037    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:47:30.060475 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:30.060420    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85967c5989-mn8js" podStartSLOduration=1.060403226 podStartE2EDuration="1.060403226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:30.05861769 +0000 UTC m=+124.129714914" watchObservedRunningTime="2026-09-16 18:47:30.060403226 +0000 UTC m=+124.131500450"
Sep 16 18:47:30.065001 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:30.064983    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fjkbf\""
Sep 16 18:47:30.072956 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:30.072930    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"
Sep 16 18:47:30.252767 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:30.252725    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f9dc5957d-6j7sr"]
Sep 16 18:47:30.257193 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:30.257163    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783e7bea_f22c_4f77_9958_facda1f07bd9.slice/crio-13b99f3940670e1268871485875c9f3f8dd3482e04330f9cf88d160c3e3256c5 WatchSource:0}: Error finding container 13b99f3940670e1268871485875c9f3f8dd3482e04330f9cf88d160c3e3256c5: Status 404 returned error can't find the container with id 13b99f3940670e1268871485875c9f3f8dd3482e04330f9cf88d160c3e3256c5
Sep 16 18:47:30.354016 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:30.353878    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg"]
Sep 16 18:47:30.357960 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:30.357927    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab856818_b5aa_4b90_b7ac_fc9743acf780.slice/crio-17337697f113829050921efc72558f22f75b4b5319ce915e0520df64580e3cd3 WatchSource:0}: Error finding container 17337697f113829050921efc72558f22f75b4b5319ce915e0520df64580e3cd3: Status 404 returned error can't find the container with id 17337697f113829050921efc72558f22f75b4b5319ce915e0520df64580e3cd3
Sep 16 18:47:31.039796 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:31.039759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg" event={"ID":"ab856818-b5aa-4b90-b7ac-fc9743acf780","Type":"ContainerStarted","Data":"17337697f113829050921efc72558f22f75b4b5319ce915e0520df64580e3cd3"}
Sep 16 18:47:31.041682 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:31.041645    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f9dc5957d-6j7sr" event={"ID":"783e7bea-f22c-4f77-9958-facda1f07bd9","Type":"ContainerStarted","Data":"141a489bc4e46d67d8ca9159b58bff35ed8dc0f26970e79390134fcc49a6189e"}
Sep 16 18:47:31.041841 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:31.041702    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f9dc5957d-6j7sr" event={"ID":"783e7bea-f22c-4f77-9958-facda1f07bd9","Type":"ContainerStarted","Data":"13b99f3940670e1268871485875c9f3f8dd3482e04330f9cf88d160c3e3256c5"}
Sep 16 18:47:31.068948 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:31.067912    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f9dc5957d-6j7sr" podStartSLOduration=34.067890426 podStartE2EDuration="34.067890426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:31.066760326 +0000 UTC m=+125.137857551" watchObservedRunningTime="2026-09-16 18:47:31.067890426 +0000 UTC m=+125.138987839"
Sep 16 18:47:31.969459 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:31.969418    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:31.972457 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:31.972433    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:32.045916 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:32.045885    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:32.047087 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:32.047064    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f9dc5957d-6j7sr"
Sep 16 18:47:33.050924 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:33.050888    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg" event={"ID":"ab856818-b5aa-4b90-b7ac-fc9743acf780","Type":"ContainerStarted","Data":"3c10ce20a0b2439b04c7ea6578719fcdacce17a10be260500a124aeb0be1ed7c"}
Sep 16 18:47:33.081772 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:33.081721    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-fnzwg" podStartSLOduration=34.207732575 podStartE2EDuration="36.081704218s" totalImagesPullingTime="1.873971643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:33.081609624 +0000 UTC m=+127.152706848" watchObservedRunningTime="2026-09-16 18:47:33.081704218 +0000 UTC m=+127.152801443"
Sep 16 18:47:34.618825 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:34.618778    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:34.621623 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:34.621597    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7df4f878-1807-43fd-acdd-3a6cff6bc930-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l97v6\" (UID: \"7df4f878-1807-43fd-acdd-3a6cff6bc930\") " pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:34.860894 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:34.860861    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-46pc5\""
Sep 16 18:47:34.868901 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:34.868823    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l97v6"
Sep 16 18:47:35.154122 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:35.154043    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l97v6"]
Sep 16 18:47:35.157255 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:35.157226    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df4f878_1807_43fd_acdd_3a6cff6bc930.slice/crio-3c6f3822ad4cb2f31a81f8534f8d4a659d4f5f34cbdd483ea57b053444b14e7a WatchSource:0}: Error finding container 3c6f3822ad4cb2f31a81f8534f8d4a659d4f5f34cbdd483ea57b053444b14e7a: Status 404 returned error can't find the container with id 3c6f3822ad4cb2f31a81f8534f8d4a659d4f5f34cbdd483ea57b053444b14e7a
Sep 16 18:47:36.063559 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.063453    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l97v6" event={"ID":"7df4f878-1807-43fd-acdd-3a6cff6bc930","Type":"ContainerStarted","Data":"7accdd4ad10c9bde92e8376230b956ab30a9cf3c43fc6f76731b5f8a7749d277"}
Sep 16 18:47:36.063937 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.063572    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l97v6" event={"ID":"7df4f878-1807-43fd-acdd-3a6cff6bc930","Type":"ContainerStarted","Data":"95156b0bf338336a898b798ba53a2b1d6cbe2786dd25fb351c58ef8f1d4da25e"}
Sep 16 18:47:36.063937 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.063594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l97v6" event={"ID":"7df4f878-1807-43fd-acdd-3a6cff6bc930","Type":"ContainerStarted","Data":"3c6f3822ad4cb2f31a81f8534f8d4a659d4f5f34cbdd483ea57b053444b14e7a"}
Sep 16 18:47:36.234661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.234625    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:47:36.237366 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.237336    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4cd60c-ec4c-4284-b45e-bb39c7786cf8-metrics-certs\") pod \"network-metrics-daemon-856sw\" (UID: \"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8\") " pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:47:36.356869 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.356788    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtjvb\""
Sep 16 18:47:36.364975 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.364938    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-856sw"
Sep 16 18:47:36.655596 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.655498    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-856sw"]
Sep 16 18:47:36.661311 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:36.661277    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4cd60c_ec4c_4284_b45e_bb39c7786cf8.slice/crio-c84bffcd274ff44231d4fbbfd505016201e45708c233894fdba5c251a7834ec7 WatchSource:0}: Error finding container c84bffcd274ff44231d4fbbfd505016201e45708c233894fdba5c251a7834ec7: Status 404 returned error can't find the container with id c84bffcd274ff44231d4fbbfd505016201e45708c233894fdba5c251a7834ec7
Sep 16 18:47:36.911628 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.911540    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"]
Sep 16 18:47:36.915639 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.915617    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:36.919309 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.919286    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:47:36.919707 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.919540    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:47:36.919707 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.919549    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7v7dh\""
Sep 16 18:47:36.919707 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.919549    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:47:36.929778 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.929756    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"]
Sep 16 18:47:36.939779 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.939755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t97hb\" (UniqueName: \"kubernetes.io/projected/f68efdfd-99b2-4954-9ed3-d295110703b7-kube-api-access-t97hb\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:36.939874 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.939798    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f68efdfd-99b2-4954-9ed3-d295110703b7-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:36.939874 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.939818    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f68efdfd-99b2-4954-9ed3-d295110703b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:36.939874 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:36.939837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68efdfd-99b2-4954-9ed3-d295110703b7-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.040457 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.040418    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t97hb\" (UniqueName: \"kubernetes.io/projected/f68efdfd-99b2-4954-9ed3-d295110703b7-kube-api-access-t97hb\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.040678 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.040470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f68efdfd-99b2-4954-9ed3-d295110703b7-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.040678 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.040503    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f68efdfd-99b2-4954-9ed3-d295110703b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.040678 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.040547    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68efdfd-99b2-4954-9ed3-d295110703b7-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.041418 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.041386    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f68efdfd-99b2-4954-9ed3-d295110703b7-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.044025 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.043981    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f68efdfd-99b2-4954-9ed3-d295110703b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.044129 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.044112    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68efdfd-99b2-4954-9ed3-d295110703b7-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.052188 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.052156    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t97hb\" (UniqueName: \"kubernetes.io/projected/f68efdfd-99b2-4954-9ed3-d295110703b7-kube-api-access-t97hb\") pod \"prometheus-operator-68bcd7f987-wvnxt\" (UID: \"f68efdfd-99b2-4954-9ed3-d295110703b7\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.068835 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.068793    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-856sw" event={"ID":"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8","Type":"ContainerStarted","Data":"c84bffcd274ff44231d4fbbfd505016201e45708c233894fdba5c251a7834ec7"}
Sep 16 18:47:37.228444 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.228411    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"
Sep 16 18:47:37.535969 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:37.535875    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt"]
Sep 16 18:47:37.542052 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:37.542027    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68efdfd_99b2_4954_9ed3_d295110703b7.slice/crio-1e4718c9f86074228bfad245f8466ed48521cd3a4b98e542be7c415a68b55980 WatchSource:0}: Error finding container 1e4718c9f86074228bfad245f8466ed48521cd3a4b98e542be7c415a68b55980: Status 404 returned error can't find the container with id 1e4718c9f86074228bfad245f8466ed48521cd3a4b98e542be7c415a68b55980
Sep 16 18:47:38.077756 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:38.077718    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l97v6" event={"ID":"7df4f878-1807-43fd-acdd-3a6cff6bc930","Type":"ContainerStarted","Data":"dd2906d03aad0510a1c75b73353d89c5ed4ff8f434abd83a4483a9a70a1c1654"}
Sep 16 18:47:38.080037 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:38.080011    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-856sw" event={"ID":"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8","Type":"ContainerStarted","Data":"0678d19cde2b041c4bc94f4e63b84ef9d5e05f0f67f5a53300b996f741e64772"}
Sep 16 18:47:38.086541 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:38.084046    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt" event={"ID":"f68efdfd-99b2-4954-9ed3-d295110703b7","Type":"ContainerStarted","Data":"1e4718c9f86074228bfad245f8466ed48521cd3a4b98e542be7c415a68b55980"}
Sep 16 18:47:38.126400 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:38.126342    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l97v6" podStartSLOduration=34.191981674 podStartE2EDuration="36.126320287s" totalImagesPullingTime="1.934338613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:38.124100619 +0000 UTC m=+132.195197844" watchObservedRunningTime="2026-09-16 18:47:38.126320287 +0000 UTC m=+132.197417511"
Sep 16 18:47:39.094973 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:39.094885    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-856sw" event={"ID":"2c4cd60c-ec4c-4284-b45e-bb39c7786cf8","Type":"ContainerStarted","Data":"5ea60dd27e6ef9c8017a9737a25a0b9ea26080ff5a8a9e8ee7a720d68a3b13ee"}
Sep 16 18:47:39.097187 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:39.097149    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt" event={"ID":"f68efdfd-99b2-4954-9ed3-d295110703b7","Type":"ContainerStarted","Data":"65ee8c8922db44c34a04080271c6b9bad286a8647e06b32bfe046f963054bf38"}
Sep 16 18:47:39.097375 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:39.097214    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt" event={"ID":"f68efdfd-99b2-4954-9ed3-d295110703b7","Type":"ContainerStarted","Data":"7fee9f83da6b490bd84065bb76ea14c93b2c07b79b0fc162facb34dbd7fda298"}
Sep 16 18:47:39.173417 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:39.172936    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-856sw" podStartSLOduration=130.935849688 podStartE2EDuration="2m12.172917518s" totalImagesPullingTime="1.23706783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:39.123459675 +0000 UTC m=+133.194556899" watchObservedRunningTime="2026-09-16 18:47:39.172917518 +0000 UTC m=+133.244014743"
Sep 16 18:47:39.173417 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:39.173159    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wvnxt" podStartSLOduration=1.937341416 podStartE2EDuration="3.173151886s" totalImagesPullingTime="1.23581047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:39.171745254 +0000 UTC m=+133.242842481" watchObservedRunningTime="2026-09-16 18:47:39.173151886 +0000 UTC m=+133.244249108"
Sep 16 18:47:41.635183 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.635148    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7xcg9"]
Sep 16 18:47:41.639807 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.639777    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.652185 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.652162    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:47:41.652319 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.652285    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vbsp9\""
Sep 16 18:47:41.652444 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.652422    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:47:41.652797 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.652777    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:47:41.683007 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.682967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42466999-4bb2-480c-941b-6365812b70f2-sys\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.683151 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.683015    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42466999-4bb2-480c-941b-6365812b70f2-metrics-client-ca\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.683151 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.683043    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.683151 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.683108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-tls\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.683151 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.683135    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-wtmp\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.683339 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.683173    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7s55\" (UniqueName: \"kubernetes.io/projected/42466999-4bb2-480c-941b-6365812b70f2-kube-api-access-p7s55\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.683339 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.683208    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42466999-4bb2-480c-941b-6365812b70f2-root\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.683339 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.683232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-textfile\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.683339 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.683250    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.783909 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.783870    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-tls\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784099 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.783920    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-wtmp\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784099 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.783963    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7s55\" (UniqueName: \"kubernetes.io/projected/42466999-4bb2-480c-941b-6365812b70f2-kube-api-access-p7s55\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784099 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.783994    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42466999-4bb2-480c-941b-6365812b70f2-root\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784099 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:41.783998    2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:47:41.784099 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-textfile\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784099 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784046    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784099 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:41.784073    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-tls podName:42466999-4bb2-480c-941b-6365812b70f2 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:42.284052654 +0000 UTC m=+136.355149875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-tls") pod "node-exporter-7xcg9" (UID: "42466999-4bb2-480c-941b-6365812b70f2") : secret "node-exporter-tls" not found
Sep 16 18:47:41.784405 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784099    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42466999-4bb2-480c-941b-6365812b70f2-root\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784405 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784122    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-wtmp\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784405 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784171    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42466999-4bb2-480c-941b-6365812b70f2-sys\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784405 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784204    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42466999-4bb2-480c-941b-6365812b70f2-metrics-client-ca\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784405 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784229    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784405 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784254    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42466999-4bb2-480c-941b-6365812b70f2-sys\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784405 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784374    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-textfile\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784722 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784702    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.784800 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.784720    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42466999-4bb2-480c-941b-6365812b70f2-metrics-client-ca\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.786921 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.786896    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:41.809440 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:41.809411    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7s55\" (UniqueName: \"kubernetes.io/projected/42466999-4bb2-480c-941b-6365812b70f2-kube-api-access-p7s55\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:42.288747 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:42.288707    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-tls\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:42.291817 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:42.291757    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42466999-4bb2-480c-941b-6365812b70f2-node-exporter-tls\") pod \"node-exporter-7xcg9\" (UID: \"42466999-4bb2-480c-941b-6365812b70f2\") " pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:42.550034 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:42.549938    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7xcg9"
Sep 16 18:47:44.391465 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.391431    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76664c6f65-rw2nd"]
Sep 16 18:47:44.396817 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.396793    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.399807 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.399785    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:47:44.399807 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.399796    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:47:44.403124 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.403033    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:47:44.403124 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.403087    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:47:44.403386 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.403366    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jk5cr\""
Sep 16 18:47:44.403542 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.403368    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bdepasv0u2u6q\""
Sep 16 18:47:44.403643 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.403564    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:47:44.418634 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.418551    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76664c6f65-rw2nd"]
Sep 16 18:47:44.509270 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.509231    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-grpc-tls\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.509446 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.509272    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.509446 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.509299    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lksrl\" (UniqueName: \"kubernetes.io/projected/64b66e25-8961-4710-b06a-04382d3fb8e9-kube-api-access-lksrl\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.509446 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.509327    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64b66e25-8961-4710-b06a-04382d3fb8e9-metrics-client-ca\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.509446 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.509388    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.509655 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.509453    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.509655 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.509574    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.509655 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.509612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-tls\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.610358 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.610303    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.610570 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.610367    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-tls\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.610570 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.610409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-grpc-tls\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.610570 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.610435    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.610570 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.610459    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksrl\" (UniqueName: \"kubernetes.io/projected/64b66e25-8961-4710-b06a-04382d3fb8e9-kube-api-access-lksrl\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.610570 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.610487    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64b66e25-8961-4710-b06a-04382d3fb8e9-metrics-client-ca\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.610570 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.610530    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.610570 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.610569    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.612026 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.611969    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64b66e25-8961-4710-b06a-04382d3fb8e9-metrics-client-ca\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.614072 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.614021    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.614595 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.614571    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-grpc-tls\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.614679 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.614630    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.614679 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.614665    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.614958 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.614921    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-tls\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.615479 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.615448    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/64b66e25-8961-4710-b06a-04382d3fb8e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.624398 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.624376    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksrl\" (UniqueName: \"kubernetes.io/projected/64b66e25-8961-4710-b06a-04382d3fb8e9-kube-api-access-lksrl\") pod \"thanos-querier-76664c6f65-rw2nd\" (UID: \"64b66e25-8961-4710-b06a-04382d3fb8e9\") " pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:44.708946 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:44.708633    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:46.111917 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.111874    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"]
Sep 16 18:47:46.115143 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.115120    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:46.118618 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.118587    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2b5lr\""
Sep 16 18:47:46.118849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.118705    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:47:46.131937 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.131905    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"]
Sep 16 18:47:46.224999 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.224961    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4119327c-86c6-4e55-b4e5-9c47b4219fd2-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-jmmcz\" (UID: \"4119327c-86c6-4e55-b4e5-9c47b4219fd2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:46.225185 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.225121    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp77p\" (UniqueName: \"kubernetes.io/projected/4119327c-86c6-4e55-b4e5-9c47b4219fd2-kube-api-access-hp77p\") pod \"monitoring-plugin-5d85778b9d-jmmcz\" (UID: \"4119327c-86c6-4e55-b4e5-9c47b4219fd2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:46.326073 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.326030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4119327c-86c6-4e55-b4e5-9c47b4219fd2-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-jmmcz\" (UID: \"4119327c-86c6-4e55-b4e5-9c47b4219fd2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:46.326256 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.326143    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp77p\" (UniqueName: \"kubernetes.io/projected/4119327c-86c6-4e55-b4e5-9c47b4219fd2-kube-api-access-hp77p\") pod \"monitoring-plugin-5d85778b9d-jmmcz\" (UID: \"4119327c-86c6-4e55-b4e5-9c47b4219fd2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:46.326256 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:46.326222    2572 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 18:47:46.326347 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:47:46.326310    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4119327c-86c6-4e55-b4e5-9c47b4219fd2-monitoring-plugin-cert podName:4119327c-86c6-4e55-b4e5-9c47b4219fd2 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:46.826287303 +0000 UTC m=+140.897384514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4119327c-86c6-4e55-b4e5-9c47b4219fd2-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-jmmcz" (UID: "4119327c-86c6-4e55-b4e5-9c47b4219fd2") : secret "monitoring-plugin-cert" not found
Sep 16 18:47:46.367410 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.367324    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp77p\" (UniqueName: \"kubernetes.io/projected/4119327c-86c6-4e55-b4e5-9c47b4219fd2-kube-api-access-hp77p\") pod \"monitoring-plugin-5d85778b9d-jmmcz\" (UID: \"4119327c-86c6-4e55-b4e5-9c47b4219fd2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:46.831359 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.831318    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4119327c-86c6-4e55-b4e5-9c47b4219fd2-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-jmmcz\" (UID: \"4119327c-86c6-4e55-b4e5-9c47b4219fd2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:46.834118 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:46.834090    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4119327c-86c6-4e55-b4e5-9c47b4219fd2-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-jmmcz\" (UID: \"4119327c-86c6-4e55-b4e5-9c47b4219fd2\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:47.027044 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:47.026991    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:47.581550 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:47.581399    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"]
Sep 16 18:47:47.585119 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:47.585082    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4119327c_86c6_4e55_b4e5_9c47b4219fd2.slice/crio-562d03d254860c8b346ba8e209e11d4bf8e080a3013482195183d8b42dc43923 WatchSource:0}: Error finding container 562d03d254860c8b346ba8e209e11d4bf8e080a3013482195183d8b42dc43923: Status 404 returned error can't find the container with id 562d03d254860c8b346ba8e209e11d4bf8e080a3013482195183d8b42dc43923
Sep 16 18:47:47.585264 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:47.585235    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76664c6f65-rw2nd"]
Sep 16 18:47:47.598666 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:47:47.598631    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b66e25_8961_4710_b06a_04382d3fb8e9.slice/crio-419c8afb580983039cf9431563d1cdae535b9f73dd78da44674bbca5b44d3345 WatchSource:0}: Error finding container 419c8afb580983039cf9431563d1cdae535b9f73dd78da44674bbca5b44d3345: Status 404 returned error can't find the container with id 419c8afb580983039cf9431563d1cdae535b9f73dd78da44674bbca5b44d3345
Sep 16 18:47:48.134610 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:48.134571    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-95c57" event={"ID":"d1108c81-4a31-4a0f-8856-419c30c68edb","Type":"ContainerStarted","Data":"d37d1d373a35b753ae886a6c64c4fd5a08a73c0f25d876564ef933334ba002f6"}
Sep 16 18:47:48.135601 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:48.135576    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-95c57"
Sep 16 18:47:48.140210 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:48.140107    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd" event={"ID":"64b66e25-8961-4710-b06a-04382d3fb8e9","Type":"ContainerStarted","Data":"419c8afb580983039cf9431563d1cdae535b9f73dd78da44674bbca5b44d3345"}
Sep 16 18:47:48.142176 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:48.142150    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz" event={"ID":"4119327c-86c6-4e55-b4e5-9c47b4219fd2","Type":"ContainerStarted","Data":"562d03d254860c8b346ba8e209e11d4bf8e080a3013482195183d8b42dc43923"}
Sep 16 18:47:48.145893 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:48.145824    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7xcg9" event={"ID":"42466999-4bb2-480c-941b-6365812b70f2","Type":"ContainerStarted","Data":"b2134368ce23800d75c3a720c93415160fbf2e921f99f6021ea47fcc18235566"}
Sep 16 18:47:48.148634 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:48.148589    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-95c57"
Sep 16 18:47:48.197105 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:48.195161    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-95c57" podStartSLOduration=1.6672279140000001 podStartE2EDuration="19.19513764s" totalImagesPullingTime="17.527909726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:48.192958236 +0000 UTC m=+142.264055461" watchObservedRunningTime="2026-09-16 18:47:48.19513764 +0000 UTC m=+142.266234863"
Sep 16 18:47:49.154364 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:49.154281    2572 generic.go:356] "Generic (PLEG): container finished" podID="42466999-4bb2-480c-941b-6365812b70f2" containerID="0519e5aa741f282b8999b420c7a0387f8c403b92220a75305fa6f3480dbfda6b" exitCode=0
Sep 16 18:47:49.155222 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:49.155196    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7xcg9" event={"ID":"42466999-4bb2-480c-941b-6365812b70f2","Type":"ContainerDied","Data":"0519e5aa741f282b8999b420c7a0387f8c403b92220a75305fa6f3480dbfda6b"}
Sep 16 18:47:49.662890 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:49.662835    2572 patch_prober.go:28] interesting pod/image-registry-85967c5989-mn8js container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:47:49.663064 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:49.662906    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85967c5989-mn8js" podUID="5d793038-dc8f-45d8-a3b9-af84ae589c23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:47:51.047626 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.047586    2572 patch_prober.go:28] interesting pod/image-registry-85967c5989-mn8js container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:47:51.048008 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.047648    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85967c5989-mn8js" podUID="5d793038-dc8f-45d8-a3b9-af84ae589c23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:47:51.168557 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.168448    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd" event={"ID":"64b66e25-8961-4710-b06a-04382d3fb8e9","Type":"ContainerStarted","Data":"9b40a1fcf0447a9fcd904d9b8d4dab96dd1851671daeac30eb19d875c2245664"}
Sep 16 18:47:51.168557 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.168551    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd" event={"ID":"64b66e25-8961-4710-b06a-04382d3fb8e9","Type":"ContainerStarted","Data":"96d2dc28db8ce3d8048d7c1595fa98e4756169b0e9d0acc9e71d70a52d9880e9"}
Sep 16 18:47:51.168822 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.168594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd" event={"ID":"64b66e25-8961-4710-b06a-04382d3fb8e9","Type":"ContainerStarted","Data":"1713065fa1ffe4903245db3ca25d1fdc8438cb1c89efc448d1953f524b4c7b0b"}
Sep 16 18:47:51.170353 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.170320    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz" event={"ID":"4119327c-86c6-4e55-b4e5-9c47b4219fd2","Type":"ContainerStarted","Data":"2dfec0484d738bdd1196ac1d6ad8407335f235751ba733d23f6aa82d75a79a86"}
Sep 16 18:47:51.170557 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.170538    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:51.173456 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.173427    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7xcg9" event={"ID":"42466999-4bb2-480c-941b-6365812b70f2","Type":"ContainerStarted","Data":"2e0fd1922655412c3085086bd58c52f01f463050a92734a27d8c657c7e458618"}
Sep 16 18:47:51.173593 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.173478    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7xcg9" event={"ID":"42466999-4bb2-480c-941b-6365812b70f2","Type":"ContainerStarted","Data":"79d96251e435e0e959b7429988b56e187d0810d1949b7364e716c5a3241d866e"}
Sep 16 18:47:51.176194 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.176170    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz"
Sep 16 18:47:51.194286 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.194232    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-jmmcz" podStartSLOduration=2.184867771 podStartE2EDuration="5.19421698s" totalImagesPullingTime="3.009349209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:51.193995804 +0000 UTC m=+145.265093029" watchObservedRunningTime="2026-09-16 18:47:51.19421698 +0000 UTC m=+145.265314198"
Sep 16 18:47:51.240186 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:51.240126    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7xcg9" podStartSLOduration=8.739516221 podStartE2EDuration="10.240105324s" totalImagesPullingTime="1.408411882s" totalInitContainerRuntime="92.177221ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:51.220168228 +0000 UTC m=+145.291265452" watchObservedRunningTime="2026-09-16 18:47:51.240105324 +0000 UTC m=+145.311202551"
Sep 16 18:47:53.187091 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:53.187054    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd" event={"ID":"64b66e25-8961-4710-b06a-04382d3fb8e9","Type":"ContainerStarted","Data":"9fa7886578cd5ab468c6bcf5b967090ad95f42feb6b7fb959a01fa7fd3328dea"}
Sep 16 18:47:53.187619 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:53.187133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd" event={"ID":"64b66e25-8961-4710-b06a-04382d3fb8e9","Type":"ContainerStarted","Data":"e25a217b8dc9f734288fa0023e5903e62ecef8c4a79f72984125ed19d03f230d"}
Sep 16 18:47:53.187619 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:53.187183    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd" event={"ID":"64b66e25-8961-4710-b06a-04382d3fb8e9","Type":"ContainerStarted","Data":"ac7350259ef68e6104e1b4a9314767621a0c34bb74ef1af83d1008d5bd5ad468"}
Sep 16 18:47:53.222609 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:53.222486    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd" podStartSLOduration=4.931588966 podStartE2EDuration="9.22246718s" totalImagesPullingTime="4.290878214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:53.220777975 +0000 UTC m=+147.291875235" watchObservedRunningTime="2026-09-16 18:47:53.22246718 +0000 UTC m=+147.293564401"
Sep 16 18:47:54.192032 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:54.191993    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:55.202314 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:55.202284    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76664c6f65-rw2nd"
Sep 16 18:47:59.660743 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:59.660712    2572 patch_prober.go:28] interesting pod/image-registry-85967c5989-mn8js container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:47:59.661231 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:47:59.660765    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85967c5989-mn8js" podUID="5d793038-dc8f-45d8-a3b9-af84ae589c23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:48:01.045936 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:01.045899    2572 patch_prober.go:28] interesting pod/image-registry-85967c5989-mn8js container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:48:01.046321 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:01.045965    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85967c5989-mn8js" podUID="5d793038-dc8f-45d8-a3b9-af84ae589c23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:48:04.858135 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:48:04.858083    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zp57f" podUID="b713bc4f-84ae-422e-8c29-0e37ac9f5f8d"
Sep 16 18:48:05.233349 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:05.233314    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp57f"
Sep 16 18:48:09.660411 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:09.660377    2572 patch_prober.go:28] interesting pod/image-registry-85967c5989-mn8js container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:48:09.660946 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:09.660436    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85967c5989-mn8js" podUID="5d793038-dc8f-45d8-a3b9-af84ae589c23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:48:09.660946 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:09.660489    2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:48:09.661125 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:09.661091    2572 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"1d9214b31d2a315889b46c023ea71e5e8572124b900336d25e5c63ab8c05e2d7"} pod="openshift-image-registry/image-registry-85967c5989-mn8js" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 16 18:48:09.664793 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:09.664768    2572 patch_prober.go:28] interesting pod/image-registry-85967c5989-mn8js container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:48:09.664907 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:09.664809    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85967c5989-mn8js" podUID="5d793038-dc8f-45d8-a3b9-af84ae589c23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:48:09.742767 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:09.742723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:48:09.745154 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:09.745128    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b713bc4f-84ae-422e-8c29-0e37ac9f5f8d-metrics-tls\") pod \"dns-default-zp57f\" (UID: \"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d\") " pod="openshift-dns/dns-default-zp57f"
Sep 16 18:48:10.036777 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:10.036744    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8559\""
Sep 16 18:48:10.044690 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:10.044666    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp57f"
Sep 16 18:48:10.321742 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:10.321626    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zp57f"]
Sep 16 18:48:10.324806 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:48:10.324776    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb713bc4f_84ae_422e_8c29_0e37ac9f5f8d.slice/crio-5c8b042a0d169a055dfa2478a146b723378d697c25c9cc88e02422dfede83c50 WatchSource:0}: Error finding container 5c8b042a0d169a055dfa2478a146b723378d697c25c9cc88e02422dfede83c50: Status 404 returned error can't find the container with id 5c8b042a0d169a055dfa2478a146b723378d697c25c9cc88e02422dfede83c50
Sep 16 18:48:11.257032 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:11.256985    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp57f" event={"ID":"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d","Type":"ContainerStarted","Data":"5c8b042a0d169a055dfa2478a146b723378d697c25c9cc88e02422dfede83c50"}
Sep 16 18:48:12.263179 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:12.263145    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp57f" event={"ID":"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d","Type":"ContainerStarted","Data":"51be97a91c4449735fbfb10077e34a4bd8df271b942749cdd93e89e06e31ccd1"}
Sep 16 18:48:12.263610 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:12.263224    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp57f" event={"ID":"b713bc4f-84ae-422e-8c29-0e37ac9f5f8d","Type":"ContainerStarted","Data":"022e25703f6d9a8827159a271840df059ac5f420542791141413408e71b36d45"}
Sep 16 18:48:12.263610 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:12.263268    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zp57f"
Sep 16 18:48:12.285419 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:12.285364    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zp57f" podStartSLOduration=129.803065493 podStartE2EDuration="2m11.285337434s" totalImagesPullingTime="1.482271941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:12.285158991 +0000 UTC m=+166.356256214" watchObservedRunningTime="2026-09-16 18:48:12.285337434 +0000 UTC m=+166.356434657"
Sep 16 18:48:19.664658 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:19.664579    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:48:22.269653 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:22.269625    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zp57f"
Sep 16 18:48:22.301488 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:22.301453    2572 generic.go:356] "Generic (PLEG): container finished" podID="3c09636a-36dd-41e9-a35b-41831a48fc4a" containerID="602748e9543276b8371c5bff942db7437aa5d19516e79f97b7738106fe70b633" exitCode=0
Sep 16 18:48:22.301488 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:22.301475    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5" event={"ID":"3c09636a-36dd-41e9-a35b-41831a48fc4a","Type":"ContainerDied","Data":"602748e9543276b8371c5bff942db7437aa5d19516e79f97b7738106fe70b633"}
Sep 16 18:48:22.301887 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:22.301851    2572 scope.go:117] "RemoveContainer" containerID="602748e9543276b8371c5bff942db7437aa5d19516e79f97b7738106fe70b633"
Sep 16 18:48:22.303381 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:22.303356    2572 generic.go:356] "Generic (PLEG): container finished" podID="71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff" containerID="4d34aa761c58c6829bedf5806795f80047b2058b5b0efd28cff68dafe667cf63" exitCode=0
Sep 16 18:48:22.303501 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:22.303411    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t" event={"ID":"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff","Type":"ContainerDied","Data":"4d34aa761c58c6829bedf5806795f80047b2058b5b0efd28cff68dafe667cf63"}
Sep 16 18:48:22.303717 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:22.303700    2572 scope.go:117] "RemoveContainer" containerID="4d34aa761c58c6829bedf5806795f80047b2058b5b0efd28cff68dafe667cf63"
Sep 16 18:48:23.313826 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:23.313788    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2xpx5" event={"ID":"3c09636a-36dd-41e9-a35b-41831a48fc4a","Type":"ContainerStarted","Data":"c821af789235341715ffe03d3b77edd2853bcbae3b2fffce72542350a22649f6"}
Sep 16 18:48:23.315605 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:23.315584    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n825t" event={"ID":"71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff","Type":"ContainerStarted","Data":"293de7fbd0e23709c16bac45cae6b4945b653921dad94587f7e9fee9efc20ce7"}
Sep 16 18:48:34.680406 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:34.680351    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85967c5989-mn8js" podUID="5d793038-dc8f-45d8-a3b9-af84ae589c23" containerName="registry" containerID="cri-o://1d9214b31d2a315889b46c023ea71e5e8572124b900336d25e5c63ab8c05e2d7" gracePeriod=30
Sep 16 18:48:36.368085 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:36.368050    2572 generic.go:356] "Generic (PLEG): container finished" podID="5d793038-dc8f-45d8-a3b9-af84ae589c23" containerID="1d9214b31d2a315889b46c023ea71e5e8572124b900336d25e5c63ab8c05e2d7" exitCode=0
Sep 16 18:48:36.368477 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:36.368098    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85967c5989-mn8js" event={"ID":"5d793038-dc8f-45d8-a3b9-af84ae589c23","Type":"ContainerDied","Data":"1d9214b31d2a315889b46c023ea71e5e8572124b900336d25e5c63ab8c05e2d7"}
Sep 16 18:48:36.368477 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:36.368176    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85967c5989-mn8js" event={"ID":"5d793038-dc8f-45d8-a3b9-af84ae589c23","Type":"ContainerStarted","Data":"f591ba9323c132e724fa22fe6679a60beb71b2830a4df6e9d5a167ea9c8f96f3"}
Sep 16 18:48:36.368477 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:36.368196    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:48:57.375861 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:48:57.375831    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85967c5989-mn8js"
Sep 16 18:50:26.354620 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:50:26.354586    2572 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:51:30.726449 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.726373    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"]
Sep 16 18:51:30.729642 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.729625    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:30.731808 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.731787    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:51:30.732234 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.732218    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xhth4\""
Sep 16 18:51:30.732336 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.732252    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:51:30.739283 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.739256    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"]
Sep 16 18:51:30.763563 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.763529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s54sr\" (UniqueName: \"kubernetes.io/projected/ef86bb07-0340-4888-8085-10ffe7636d1f-kube-api-access-s54sr\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:30.763732 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.763571    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:30.763732 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.763660    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:30.864129 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.864093    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s54sr\" (UniqueName: \"kubernetes.io/projected/ef86bb07-0340-4888-8085-10ffe7636d1f-kube-api-access-s54sr\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:30.864321 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.864147    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:30.864321 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.864210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:30.864604 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.864585    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:30.864681 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.864605    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:30.873242 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:30.873219    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s54sr\" (UniqueName: \"kubernetes.io/projected/ef86bb07-0340-4888-8085-10ffe7636d1f-kube-api-access-s54sr\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:31.039142 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:31.039048    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:31.320454 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:31.320426    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"]
Sep 16 18:51:31.325939 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:31.325919    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:51:32.028567 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:32.028504    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h" event={"ID":"ef86bb07-0340-4888-8085-10ffe7636d1f","Type":"ContainerStarted","Data":"e28efa11ca053a84cf88d5a122b1cb72bd181b3291a6b12aecc8302f7d0efaa0"}
Sep 16 18:51:38.053808 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:38.053774    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerID="70f5578d13e530d8c8baf93346f3985c6b723f9b53dd25e4960d3312c1a838f9" exitCode=0
Sep 16 18:51:38.054256 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:38.053862    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h" event={"ID":"ef86bb07-0340-4888-8085-10ffe7636d1f","Type":"ContainerDied","Data":"70f5578d13e530d8c8baf93346f3985c6b723f9b53dd25e4960d3312c1a838f9"}
Sep 16 18:51:41.068869 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:41.068830    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerID="7f382ec71ef6e3833a160d12864ae02808c7d1f9dc766c2940b1e78e825a67f0" exitCode=0
Sep 16 18:51:41.069290 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:41.068888    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h" event={"ID":"ef86bb07-0340-4888-8085-10ffe7636d1f","Type":"ContainerDied","Data":"7f382ec71ef6e3833a160d12864ae02808c7d1f9dc766c2940b1e78e825a67f0"}
Sep 16 18:51:48.101989 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:48.101956    2572 generic.go:356] "Generic (PLEG): container finished" podID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerID="53b5e012de29bf45221c6356c638b9394aaf0e62a8f43ae0955bf2eaf6afbb26" exitCode=0
Sep 16 18:51:48.102379 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:48.102031    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h" event={"ID":"ef86bb07-0340-4888-8085-10ffe7636d1f","Type":"ContainerDied","Data":"53b5e012de29bf45221c6356c638b9394aaf0e62a8f43ae0955bf2eaf6afbb26"}
Sep 16 18:51:49.232685 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.232650    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:49.320010 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.319955    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s54sr\" (UniqueName: \"kubernetes.io/projected/ef86bb07-0340-4888-8085-10ffe7636d1f-kube-api-access-s54sr\") pod \"ef86bb07-0340-4888-8085-10ffe7636d1f\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") "
Sep 16 18:51:49.320199 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.320053    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-util\") pod \"ef86bb07-0340-4888-8085-10ffe7636d1f\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") "
Sep 16 18:51:49.320199 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.320086    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-bundle\") pod \"ef86bb07-0340-4888-8085-10ffe7636d1f\" (UID: \"ef86bb07-0340-4888-8085-10ffe7636d1f\") "
Sep 16 18:51:49.320794 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.320771    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-bundle" (OuterVolumeSpecName: "bundle") pod "ef86bb07-0340-4888-8085-10ffe7636d1f" (UID: "ef86bb07-0340-4888-8085-10ffe7636d1f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:49.322366 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.322343    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef86bb07-0340-4888-8085-10ffe7636d1f-kube-api-access-s54sr" (OuterVolumeSpecName: "kube-api-access-s54sr") pod "ef86bb07-0340-4888-8085-10ffe7636d1f" (UID: "ef86bb07-0340-4888-8085-10ffe7636d1f"). InnerVolumeSpecName "kube-api-access-s54sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:49.324845 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.324824    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-util" (OuterVolumeSpecName: "util") pod "ef86bb07-0340-4888-8085-10ffe7636d1f" (UID: "ef86bb07-0340-4888-8085-10ffe7636d1f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:49.421273 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.421175    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s54sr\" (UniqueName: \"kubernetes.io/projected/ef86bb07-0340-4888-8085-10ffe7636d1f-kube-api-access-s54sr\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 18:51:49.421273 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.421218    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-util\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 18:51:49.421273 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:49.421233    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef86bb07-0340-4888-8085-10ffe7636d1f-bundle\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 18:51:50.111664 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:50.111629    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h" event={"ID":"ef86bb07-0340-4888-8085-10ffe7636d1f","Type":"ContainerDied","Data":"e28efa11ca053a84cf88d5a122b1cb72bd181b3291a6b12aecc8302f7d0efaa0"}
Sep 16 18:51:50.111664 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:50.111648    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1avqr6h"
Sep 16 18:51:50.111868 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:50.111673    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e28efa11ca053a84cf88d5a122b1cb72bd181b3291a6b12aecc8302f7d0efaa0"
Sep 16 18:51:54.087418 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.087386    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"]
Sep 16 18:51:54.087849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.087671    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerName="util"
Sep 16 18:51:54.087849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.087683    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerName="util"
Sep 16 18:51:54.087849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.087702    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerName="extract"
Sep 16 18:51:54.087849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.087707    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerName="extract"
Sep 16 18:51:54.087849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.087718    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerName="pull"
Sep 16 18:51:54.087849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.087724    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerName="pull"
Sep 16 18:51:54.087849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.087785    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef86bb07-0340-4888-8085-10ffe7636d1f" containerName="extract"
Sep 16 18:51:54.091275 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.091258    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:51:54.093366 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.093348    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:51:54.093920 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.093896    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:51:54.094122 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.093956    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-wblr7\""
Sep 16 18:51:54.096010 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.095994    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:51:54.108438 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.108416    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"]
Sep 16 18:51:54.159750 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.159712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj8gb\" (UniqueName: \"kubernetes.io/projected/8be306a8-4a93-4cd0-b9cc-c04e3fc7f641-kube-api-access-kj8gb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fp6rd\" (UID: \"8be306a8-4a93-4cd0-b9cc-c04e3fc7f641\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:51:54.159942 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.159771    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8be306a8-4a93-4cd0-b9cc-c04e3fc7f641-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fp6rd\" (UID: \"8be306a8-4a93-4cd0-b9cc-c04e3fc7f641\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:51:54.260260 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.260226    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj8gb\" (UniqueName: \"kubernetes.io/projected/8be306a8-4a93-4cd0-b9cc-c04e3fc7f641-kube-api-access-kj8gb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fp6rd\" (UID: \"8be306a8-4a93-4cd0-b9cc-c04e3fc7f641\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:51:54.260388 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.260275    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8be306a8-4a93-4cd0-b9cc-c04e3fc7f641-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fp6rd\" (UID: \"8be306a8-4a93-4cd0-b9cc-c04e3fc7f641\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:51:54.262828 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.262807    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8be306a8-4a93-4cd0-b9cc-c04e3fc7f641-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fp6rd\" (UID: \"8be306a8-4a93-4cd0-b9cc-c04e3fc7f641\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:51:54.268927 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.268905    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj8gb\" (UniqueName: \"kubernetes.io/projected/8be306a8-4a93-4cd0-b9cc-c04e3fc7f641-kube-api-access-kj8gb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fp6rd\" (UID: \"8be306a8-4a93-4cd0-b9cc-c04e3fc7f641\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:51:54.401712 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.401626    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:51:54.681992 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:54.681715    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"]
Sep 16 18:51:54.684791 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:51:54.684756    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be306a8_4a93_4cd0_b9cc_c04e3fc7f641.slice/crio-af0d958afa0171f18c9933b88a9d9248fe90c027383c63e4df1fa201abdc805a WatchSource:0}: Error finding container af0d958afa0171f18c9933b88a9d9248fe90c027383c63e4df1fa201abdc805a: Status 404 returned error can't find the container with id af0d958afa0171f18c9933b88a9d9248fe90c027383c63e4df1fa201abdc805a
Sep 16 18:51:55.131581 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:55.131551    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd" event={"ID":"8be306a8-4a93-4cd0-b9cc-c04e3fc7f641","Type":"ContainerStarted","Data":"af0d958afa0171f18c9933b88a9d9248fe90c027383c63e4df1fa201abdc805a"}
Sep 16 18:51:58.147219 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:58.147174    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd" event={"ID":"8be306a8-4a93-4cd0-b9cc-c04e3fc7f641","Type":"ContainerStarted","Data":"91ddf0859085aa566df8cf83188c8b47f86d9d0afb07963f66fa84d0743d90f2"}
Sep 16 18:51:58.147644 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:58.147319    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:51:58.982555 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:58.982484    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd" podStartSLOduration=1.764752937 podStartE2EDuration="4.982468249s" totalImagesPullingTime="3.217715312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:58.189583602 +0000 UTC m=+392.260680826" watchObservedRunningTime="2026-09-16 18:51:58.982468249 +0000 UTC m=+393.053565470"
Sep 16 18:51:58.983171 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:58.983154    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-nh5rn"]
Sep 16 18:51:58.986419 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:58.986405    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:58.989757 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:58.989730    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:51:58.989993 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:58.989978    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:51:58.990224 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:58.990208    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-ztfjw\""
Sep 16 18:51:59.015724 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.015696    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-nh5rn"]
Sep 16 18:51:59.101393 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.101352    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgkp\" (UniqueName: \"kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-kube-api-access-gsgkp\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:59.101592 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.101409    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-certificates\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:59.101592 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.101475    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-cabundle0\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:59.202392 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.202354    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-certificates\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:59.202392 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.202407    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-cabundle0\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:59.202981 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.202458    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgkp\" (UniqueName: \"kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-kube-api-access-gsgkp\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:59.202981 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:51:59.202546    2572 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 18:51:59.202981 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:51:59.202572    2572 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:51:59.202981 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:51:59.202583    2572 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:51:59.202981 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:51:59.202598    2572 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-nh5rn: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:51:59.202981 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:51:59.202653    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-certificates podName:88ba4775-4b0e-4d5c-804b-aab5d6808cc8 nodeName:}" failed. No retries permitted until 2026-09-16 18:51:59.702635693 +0000 UTC m=+393.773732916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-certificates") pod "keda-operator-579c8958f7-nh5rn" (UID: "88ba4775-4b0e-4d5c-804b-aab5d6808cc8") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:51:59.203271 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.203108    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-cabundle0\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:59.234635 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.234571    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgkp\" (UniqueName: \"kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-kube-api-access-gsgkp\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:59.706821 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:51:59.706790    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-certificates\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:51:59.706981 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:51:59.706928    2572 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:51:59.706981 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:51:59.706943    2572 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:51:59.706981 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:51:59.706952    2572 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-nh5rn: references non-existent secret key: ca.crt
Sep 16 18:51:59.707072 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:51:59.706999    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-certificates podName:88ba4775-4b0e-4d5c-804b-aab5d6808cc8 nodeName:}" failed. No retries permitted until 2026-09-16 18:52:00.706986233 +0000 UTC m=+394.778083436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-certificates") pod "keda-operator-579c8958f7-nh5rn" (UID: "88ba4775-4b0e-4d5c-804b-aab5d6808cc8") : references non-existent secret key: ca.crt
Sep 16 18:52:00.197145 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.197119    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zfkls"]
Sep 16 18:52:00.200568 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.200548    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:52:00.207706 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.207684    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:52:00.231347 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.231321    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zfkls"]
Sep 16 18:52:00.312889 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.312847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bf74b1a-1007-41c5-99e4-532d2bd8bfe5-certificates\") pod \"keda-admission-6c4d864b86-zfkls\" (UID: \"4bf74b1a-1007-41c5-99e4-532d2bd8bfe5\") " pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:52:00.313108 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.312919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lj6t\" (UniqueName: \"kubernetes.io/projected/4bf74b1a-1007-41c5-99e4-532d2bd8bfe5-kube-api-access-5lj6t\") pod \"keda-admission-6c4d864b86-zfkls\" (UID: \"4bf74b1a-1007-41c5-99e4-532d2bd8bfe5\") " pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:52:00.413788 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.413753    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bf74b1a-1007-41c5-99e4-532d2bd8bfe5-certificates\") pod \"keda-admission-6c4d864b86-zfkls\" (UID: \"4bf74b1a-1007-41c5-99e4-532d2bd8bfe5\") " pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:52:00.413938 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.413819    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lj6t\" (UniqueName: \"kubernetes.io/projected/4bf74b1a-1007-41c5-99e4-532d2bd8bfe5-kube-api-access-5lj6t\") pod \"keda-admission-6c4d864b86-zfkls\" (UID: \"4bf74b1a-1007-41c5-99e4-532d2bd8bfe5\") " pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:52:00.416544 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.416502    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bf74b1a-1007-41c5-99e4-532d2bd8bfe5-certificates\") pod \"keda-admission-6c4d864b86-zfkls\" (UID: \"4bf74b1a-1007-41c5-99e4-532d2bd8bfe5\") " pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:52:00.433493 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.433464    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lj6t\" (UniqueName: \"kubernetes.io/projected/4bf74b1a-1007-41c5-99e4-532d2bd8bfe5-kube-api-access-5lj6t\") pod \"keda-admission-6c4d864b86-zfkls\" (UID: \"4bf74b1a-1007-41c5-99e4-532d2bd8bfe5\") " pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:52:00.510374 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.510293    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:52:00.716287 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.716248    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-certificates\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:52:00.718832 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.718814    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/88ba4775-4b0e-4d5c-804b-aab5d6808cc8-certificates\") pod \"keda-operator-579c8958f7-nh5rn\" (UID: \"88ba4775-4b0e-4d5c-804b-aab5d6808cc8\") " pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:52:00.795950 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.795865    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:52:00.810352 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:00.810325    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zfkls"]
Sep 16 18:52:00.813923 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:52:00.813897    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf74b1a_1007_41c5_99e4_532d2bd8bfe5.slice/crio-820bbda1c744a0ff6d4327451a1c15a02867b25fb8798cde245ee3acae35ba88 WatchSource:0}: Error finding container 820bbda1c744a0ff6d4327451a1c15a02867b25fb8798cde245ee3acae35ba88: Status 404 returned error can't find the container with id 820bbda1c744a0ff6d4327451a1c15a02867b25fb8798cde245ee3acae35ba88
Sep 16 18:52:01.091444 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:01.091365    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-nh5rn"]
Sep 16 18:52:01.094225 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:52:01.094200    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ba4775_4b0e_4d5c_804b_aab5d6808cc8.slice/crio-dfeea33377bf075397554f9d209bef5d101e7d89276acc4e57585bd42f71355e WatchSource:0}: Error finding container dfeea33377bf075397554f9d209bef5d101e7d89276acc4e57585bd42f71355e: Status 404 returned error can't find the container with id dfeea33377bf075397554f9d209bef5d101e7d89276acc4e57585bd42f71355e
Sep 16 18:52:01.160528 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:01.160492    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-nh5rn" event={"ID":"88ba4775-4b0e-4d5c-804b-aab5d6808cc8","Type":"ContainerStarted","Data":"dfeea33377bf075397554f9d209bef5d101e7d89276acc4e57585bd42f71355e"}
Sep 16 18:52:01.161596 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:01.161574    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zfkls" event={"ID":"4bf74b1a-1007-41c5-99e4-532d2bd8bfe5","Type":"ContainerStarted","Data":"820bbda1c744a0ff6d4327451a1c15a02867b25fb8798cde245ee3acae35ba88"}
Sep 16 18:52:05.179710 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:05.179665    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-nh5rn" event={"ID":"88ba4775-4b0e-4d5c-804b-aab5d6808cc8","Type":"ContainerStarted","Data":"b29b11eda9599c3769601c14b1f1cf26b9ddee7ae5895c86b3dabf31c3bddb8e"}
Sep 16 18:52:05.180085 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:05.179771    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:52:05.201592 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:05.201544    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-nh5rn" podStartSLOduration=3.725471634 podStartE2EDuration="7.201502124s" totalImagesPullingTime="3.47603049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:05.200797022 +0000 UTC m=+399.271894247" watchObservedRunningTime="2026-09-16 18:52:05.201502124 +0000 UTC m=+399.272599348"
Sep 16 18:52:10.201629 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:10.201594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zfkls" event={"ID":"4bf74b1a-1007-41c5-99e4-532d2bd8bfe5","Type":"ContainerStarted","Data":"cef7d12986dfb2b54ce7e7a18ac47905e3931d391491250d2d257e9a6e06b085"}
Sep 16 18:52:10.202131 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:10.201717    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:52:10.224729 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:10.224504    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-zfkls" podStartSLOduration=1.549627106 podStartE2EDuration="10.224485886s" totalImagesPullingTime="8.67485878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:10.223679433 +0000 UTC m=+404.294776657" watchObservedRunningTime="2026-09-16 18:52:10.224485886 +0000 UTC m=+404.295583112"
Sep 16 18:52:19.153700 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:19.153668    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fp6rd"
Sep 16 18:52:26.186254 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:26.186224    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-nh5rn"
Sep 16 18:52:31.209000 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:52:31.208964    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-zfkls"
Sep 16 18:53:21.161103 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.161063    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"]
Sep 16 18:53:21.164211 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.164190    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:21.166851 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.166833    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:53:21.167706 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.167679    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rt279\""
Sep 16 18:53:21.167875 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.167705    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:53:21.167875 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.167764    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:53:21.175056 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.175029    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"]
Sep 16 18:53:21.239174 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.239138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/170ed54a-d944-49a3-9023-1578e2a6fb28-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lsrhv\" (UID: \"170ed54a-d944-49a3-9023-1578e2a6fb28\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:21.239357 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.239194    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5lk\" (UniqueName: \"kubernetes.io/projected/170ed54a-d944-49a3-9023-1578e2a6fb28-kube-api-access-lh5lk\") pod \"llmisvc-controller-manager-645f9c8fd6-lsrhv\" (UID: \"170ed54a-d944-49a3-9023-1578e2a6fb28\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:21.339757 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.339712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/170ed54a-d944-49a3-9023-1578e2a6fb28-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lsrhv\" (UID: \"170ed54a-d944-49a3-9023-1578e2a6fb28\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:21.339932 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.339773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5lk\" (UniqueName: \"kubernetes.io/projected/170ed54a-d944-49a3-9023-1578e2a6fb28-kube-api-access-lh5lk\") pod \"llmisvc-controller-manager-645f9c8fd6-lsrhv\" (UID: \"170ed54a-d944-49a3-9023-1578e2a6fb28\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:21.339932 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:53:21.339876    2572 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 18:53:21.340003 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:53:21.339961    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/170ed54a-d944-49a3-9023-1578e2a6fb28-cert podName:170ed54a-d944-49a3-9023-1578e2a6fb28 nodeName:}" failed. No retries permitted until 2026-09-16 18:53:21.839938102 +0000 UTC m=+475.911035304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/170ed54a-d944-49a3-9023-1578e2a6fb28-cert") pod "llmisvc-controller-manager-645f9c8fd6-lsrhv" (UID: "170ed54a-d944-49a3-9023-1578e2a6fb28") : secret "llmisvc-webhook-server-cert" not found
Sep 16 18:53:21.352949 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.352918    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5lk\" (UniqueName: \"kubernetes.io/projected/170ed54a-d944-49a3-9023-1578e2a6fb28-kube-api-access-lh5lk\") pod \"llmisvc-controller-manager-645f9c8fd6-lsrhv\" (UID: \"170ed54a-d944-49a3-9023-1578e2a6fb28\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:21.843150 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.843109    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/170ed54a-d944-49a3-9023-1578e2a6fb28-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lsrhv\" (UID: \"170ed54a-d944-49a3-9023-1578e2a6fb28\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:21.845778 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:21.845758    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/170ed54a-d944-49a3-9023-1578e2a6fb28-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lsrhv\" (UID: \"170ed54a-d944-49a3-9023-1578e2a6fb28\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:22.075690 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:22.075643    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:22.350667 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:22.350640    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"]
Sep 16 18:53:22.352765 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:53:22.352728    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod170ed54a_d944_49a3_9023_1578e2a6fb28.slice/crio-0f7228e7e5c76f11bcb033b2075e28d1b7affe4074679eb4ec701ba3b9ac3990 WatchSource:0}: Error finding container 0f7228e7e5c76f11bcb033b2075e28d1b7affe4074679eb4ec701ba3b9ac3990: Status 404 returned error can't find the container with id 0f7228e7e5c76f11bcb033b2075e28d1b7affe4074679eb4ec701ba3b9ac3990
Sep 16 18:53:22.506173 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:22.506139    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv" event={"ID":"170ed54a-d944-49a3-9023-1578e2a6fb28","Type":"ContainerStarted","Data":"0f7228e7e5c76f11bcb033b2075e28d1b7affe4074679eb4ec701ba3b9ac3990"}
Sep 16 18:53:25.523119 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:25.523083    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv" event={"ID":"170ed54a-d944-49a3-9023-1578e2a6fb28","Type":"ContainerStarted","Data":"d4fdabbd9cf092f8c7957b9b6e43e66af96693de7110c2220fc8cc9a1251e139"}
Sep 16 18:53:25.523494 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:25.523162    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:53:25.543010 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:25.542948    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv" podStartSLOduration=2.090258712 podStartE2EDuration="4.542927077s" totalImagesPullingTime="2.452668365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:25.542509251 +0000 UTC m=+479.613606482" watchObservedRunningTime="2026-09-16 18:53:25.542927077 +0000 UTC m=+479.614024302"
Sep 16 18:53:56.529849 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:53:56.529822    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lsrhv"
Sep 16 18:54:40.057551 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.057433    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-mb9qw"]
Sep 16 18:54:40.060919 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.060897    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.063095 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.063073    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-bwrgd\""
Sep 16 18:54:40.063297 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.063279    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:54:40.063388 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.063373    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:54:40.075154 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.075130    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-mb9qw"]
Sep 16 18:54:40.169820 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.169777    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zt84\" (UniqueName: \"kubernetes.io/projected/719b95c2-5fff-4eed-87fd-716adc1339a8-kube-api-access-4zt84\") pod \"odh-model-controller-7b966d7984-mb9qw\" (UID: \"719b95c2-5fff-4eed-87fd-716adc1339a8\") " pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.169820 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.169822    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/719b95c2-5fff-4eed-87fd-716adc1339a8-cert\") pod \"odh-model-controller-7b966d7984-mb9qw\" (UID: \"719b95c2-5fff-4eed-87fd-716adc1339a8\") " pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.170043 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.169877    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719b95c2-5fff-4eed-87fd-716adc1339a8-metrics-certs\") pod \"odh-model-controller-7b966d7984-mb9qw\" (UID: \"719b95c2-5fff-4eed-87fd-716adc1339a8\") " pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.270574 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.270542    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zt84\" (UniqueName: \"kubernetes.io/projected/719b95c2-5fff-4eed-87fd-716adc1339a8-kube-api-access-4zt84\") pod \"odh-model-controller-7b966d7984-mb9qw\" (UID: \"719b95c2-5fff-4eed-87fd-716adc1339a8\") " pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.270792 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.270583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/719b95c2-5fff-4eed-87fd-716adc1339a8-cert\") pod \"odh-model-controller-7b966d7984-mb9qw\" (UID: \"719b95c2-5fff-4eed-87fd-716adc1339a8\") " pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.270792 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.270605    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719b95c2-5fff-4eed-87fd-716adc1339a8-metrics-certs\") pod \"odh-model-controller-7b966d7984-mb9qw\" (UID: \"719b95c2-5fff-4eed-87fd-716adc1339a8\") " pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.273362 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.273337    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/719b95c2-5fff-4eed-87fd-716adc1339a8-cert\") pod \"odh-model-controller-7b966d7984-mb9qw\" (UID: \"719b95c2-5fff-4eed-87fd-716adc1339a8\") " pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.273476 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.273388    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719b95c2-5fff-4eed-87fd-716adc1339a8-metrics-certs\") pod \"odh-model-controller-7b966d7984-mb9qw\" (UID: \"719b95c2-5fff-4eed-87fd-716adc1339a8\") " pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.282079 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.282057    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zt84\" (UniqueName: \"kubernetes.io/projected/719b95c2-5fff-4eed-87fd-716adc1339a8-kube-api-access-4zt84\") pod \"odh-model-controller-7b966d7984-mb9qw\" (UID: \"719b95c2-5fff-4eed-87fd-716adc1339a8\") " pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.370967 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.370882    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:40.646617 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.646593    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-mb9qw"]
Sep 16 18:54:40.648834 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:54:40.648804    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719b95c2_5fff_4eed_87fd_716adc1339a8.slice/crio-fab37a4ade7cb451d5fbd03d0cb64741bac677c0b21e7daa36c8a3dcb78bfbe5 WatchSource:0}: Error finding container fab37a4ade7cb451d5fbd03d0cb64741bac677c0b21e7daa36c8a3dcb78bfbe5: Status 404 returned error can't find the container with id fab37a4ade7cb451d5fbd03d0cb64741bac677c0b21e7daa36c8a3dcb78bfbe5
Sep 16 18:54:40.846372 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:40.846335    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-mb9qw" event={"ID":"719b95c2-5fff-4eed-87fd-716adc1339a8","Type":"ContainerStarted","Data":"fab37a4ade7cb451d5fbd03d0cb64741bac677c0b21e7daa36c8a3dcb78bfbe5"}
Sep 16 18:54:43.862925 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:43.862810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-mb9qw" event={"ID":"719b95c2-5fff-4eed-87fd-716adc1339a8","Type":"ContainerStarted","Data":"10d7e93784fba7ad1dbeb5ed4d8823bad791b96b91cb033eff64fdd7f8f03690"}
Sep 16 18:54:43.863291 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:43.862961    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:54:43.880887 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:43.880834    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-mb9qw" podStartSLOduration=1.472382385 podStartE2EDuration="3.88081591s" totalImagesPullingTime="2.408433525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:43.880334416 +0000 UTC m=+557.951431634" watchObservedRunningTime="2026-09-16 18:54:43.88081591 +0000 UTC m=+557.951913135"
Sep 16 18:54:54.869350 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:54:54.869318    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-mb9qw"
Sep 16 18:56:27.975861 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.975819    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"]
Sep 16 18:56:27.979285 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.979265    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:27.981834 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.981811    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:56:27.982066 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.982049    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:56:27.982124 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.982103    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:56:27.982178 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.982145    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:56:27.982231 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.982222    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:56:27.982472 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.982450    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:56:27.982581 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.982562    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 18:56:27.992419 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:27.992401    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"]
Sep 16 18:56:28.010726 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.010703    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.010848 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.010735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwfb\" (UniqueName: \"kubernetes.io/projected/9b635504-47bb-4126-aed6-53ac7014c69a-kube-api-access-9kwfb\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.010848 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.010778    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b635504-47bb-4126-aed6-53ac7014c69a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.010848 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.010798    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.010848 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.010844    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b635504-47bb-4126-aed6-53ac7014c69a-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.111919 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.111878    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.111919 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.111918    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwfb\" (UniqueName: \"kubernetes.io/projected/9b635504-47bb-4126-aed6-53ac7014c69a-kube-api-access-9kwfb\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.112160 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.112062    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b635504-47bb-4126-aed6-53ac7014c69a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.112160 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.112104    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.112160 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.112137    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b635504-47bb-4126-aed6-53ac7014c69a-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.112548 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.112498    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b635504-47bb-4126-aed6-53ac7014c69a-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.112657 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.112640    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.112709 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.112687    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.114756 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.114736    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b635504-47bb-4126-aed6-53ac7014c69a-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.121147 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.121122    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwfb\" (UniqueName: \"kubernetes.io/projected/9b635504-47bb-4126-aed6-53ac7014c69a-kube-api-access-9kwfb\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-ch7fv\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.289190 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.289101    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:28.564216 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:28.564178    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"]
Sep 16 18:56:28.566444 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:56:28.566413    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b635504_47bb_4126_aed6_53ac7014c69a.slice/crio-612a7550cbec961e96a309a14ccbda5e10cd9abadb5eaa0d0c3903c1fde1a3a1 WatchSource:0}: Error finding container 612a7550cbec961e96a309a14ccbda5e10cd9abadb5eaa0d0c3903c1fde1a3a1: Status 404 returned error can't find the container with id 612a7550cbec961e96a309a14ccbda5e10cd9abadb5eaa0d0c3903c1fde1a3a1
Sep 16 18:56:29.316942 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:29.316895    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" event={"ID":"9b635504-47bb-4126-aed6-53ac7014c69a","Type":"ContainerStarted","Data":"612a7550cbec961e96a309a14ccbda5e10cd9abadb5eaa0d0c3903c1fde1a3a1"}
Sep 16 18:56:32.335058 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:32.335018    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" event={"ID":"9b635504-47bb-4126-aed6-53ac7014c69a","Type":"ContainerStarted","Data":"3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56"}
Sep 16 18:56:36.353356 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:36.353317    2572 generic.go:356] "Generic (PLEG): container finished" podID="9b635504-47bb-4126-aed6-53ac7014c69a" containerID="3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56" exitCode=0
Sep 16 18:56:36.353800 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:36.353385    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" event={"ID":"9b635504-47bb-4126-aed6-53ac7014c69a","Type":"ContainerDied","Data":"3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56"}
Sep 16 18:56:36.354548 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:36.354529    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:56:50.450072 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:50.450028    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" event={"ID":"9b635504-47bb-4126-aed6-53ac7014c69a","Type":"ContainerStarted","Data":"f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d"}
Sep 16 18:56:52.456112 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:52.456078    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" event={"ID":"9b635504-47bb-4126-aed6-53ac7014c69a","Type":"ContainerStarted","Data":"9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae"}
Sep 16 18:56:52.456482 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:52.456219    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:52.486214 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:52.486159    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podStartSLOduration=2.74755854 podStartE2EDuration="25.486138148s" totalImagesPullingTime="18.889606985s" totalInitContainerRuntime="3.848972623s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:52.484725 +0000 UTC m=+686.555822247" watchObservedRunningTime="2026-09-16 18:56:52.486138148 +0000 UTC m=+686.557235374"
Sep 16 18:56:53.460733 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:53.460701    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:53.461950 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:53.461916    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:56:54.464699 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:54.464662    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:56:59.469712 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:59.469684    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:56:59.470200 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:56:59.470171    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:57:09.470766 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:57:09.470727    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:57:19.470543 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:57:19.470438    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:57:29.470661 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:57:29.470618    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:57:39.470382 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:57:39.470336    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:57:49.470934 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:57:49.470892    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:57:59.470446 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:57:59.470406    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:58:09.471626 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:09.471590    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:58:22.869362 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.869320    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"]
Sep 16 18:58:22.873369 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.873345    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.876092 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.876065    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-f7202-kube-rbac-proxy-sar-config\""
Sep 16 18:58:22.876249 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.876217    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-f7202-predictor-serving-cert\""
Sep 16 18:58:22.887027 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.886998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.887178 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.887033    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-proxy-tls\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.887178 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.887055    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-cabundle-cert\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.887178 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.887150    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.887385 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.887223    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5gj\" (UniqueName: \"kubernetes.io/projected/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-api-access-hs5gj\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.891108 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.891085    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"]
Sep 16 18:58:22.987859 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.987820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5gj\" (UniqueName: \"kubernetes.io/projected/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-api-access-hs5gj\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.988013 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.987883    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.988013 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.987987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-proxy-tls\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.988135 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.988026    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-cabundle-cert\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.988135 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.988099    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.988236 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:58:22.988102    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-serving-cert: secret "isvc-sklearn-graph-raw-f7202-predictor-serving-cert" not found
Sep 16 18:58:22.988236 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:58:22.988208    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-proxy-tls podName:76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd nodeName:}" failed. No retries permitted until 2026-09-16 18:58:23.488183321 +0000 UTC m=+777.559280534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-proxy-tls") pod "isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" (UID: "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd") : secret "isvc-sklearn-graph-raw-f7202-predictor-serving-cert" not found
Sep 16 18:58:22.988457 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.988432    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.988600 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.988583    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:22.988737 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:22.988712    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-cabundle-cert\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:23.005447 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:23.005411    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5gj\" (UniqueName: \"kubernetes.io/projected/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-api-access-hs5gj\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:23.491506 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:23.491461    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-proxy-tls\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:23.494130 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:23.494103    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-proxy-tls\") pod \"isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:23.784502 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:23.784399    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:24.059777 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:24.059685    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"]
Sep 16 18:58:24.064016 ip-10-0-128-222 kubenswrapper[2572]: W0916 18:58:24.063990    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d94e8a_c5bf_4f5a_b2a6_a2b91fc88dbd.slice/crio-4545cbfd786153ac7e783c77663e0648398e2e0ee0bbc13e6e92f0ccb7bfe9b9 WatchSource:0}: Error finding container 4545cbfd786153ac7e783c77663e0648398e2e0ee0bbc13e6e92f0ccb7bfe9b9: Status 404 returned error can't find the container with id 4545cbfd786153ac7e783c77663e0648398e2e0ee0bbc13e6e92f0ccb7bfe9b9
Sep 16 18:58:24.869250 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:24.869217    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" event={"ID":"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd","Type":"ContainerStarted","Data":"73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d"}
Sep 16 18:58:24.869450 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:24.869271    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" event={"ID":"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd","Type":"ContainerStarted","Data":"4545cbfd786153ac7e783c77663e0648398e2e0ee0bbc13e6e92f0ccb7bfe9b9"}
Sep 16 18:58:28.888094 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:28.888061    2572 generic.go:356] "Generic (PLEG): container finished" podID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerID="73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d" exitCode=0
Sep 16 18:58:28.888474 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:28.888114    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" event={"ID":"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd","Type":"ContainerDied","Data":"73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d"}
Sep 16 18:58:29.895291 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:29.895256    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" event={"ID":"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd","Type":"ContainerStarted","Data":"bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558"}
Sep 16 18:58:29.895745 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:29.895314    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" event={"ID":"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd","Type":"ContainerStarted","Data":"927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77"}
Sep 16 18:58:29.895745 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:29.895631    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:29.895832 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:29.895749    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:29.897127 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:29.897104    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 18:58:29.921459 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:29.921415    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podStartSLOduration=4.008458304 podStartE2EDuration="7.921401378s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.912943074s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:29.919168649 +0000 UTC m=+783.990265873" watchObservedRunningTime="2026-09-16 18:58:29.921401378 +0000 UTC m=+783.992498603"
Sep 16 18:58:30.899324 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:30.899289    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 18:58:35.904087 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:35.904056    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:58:35.904592 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:35.904567    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 18:58:45.904988 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:45.904942    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 18:58:55.905324 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:58:55.905244    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 18:59:05.904980 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:05.904931    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 18:59:15.905218 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:15.905169    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 18:59:25.905491 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:25.905446    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 18:59:35.904630 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:35.904582    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 18:59:45.905350 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:45.905318    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 18:59:47.679197 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:47.679162    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"]
Sep 16 18:59:47.679728 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:47.679434    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" containerID="cri-o://f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d" gracePeriod=30
Sep 16 18:59:47.679728 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:47.679477    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kube-rbac-proxy" containerID="cri-o://9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae" gracePeriod=30
Sep 16 18:59:48.259251 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:48.259220    2572 generic.go:356] "Generic (PLEG): container finished" podID="9b635504-47bb-4126-aed6-53ac7014c69a" containerID="9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae" exitCode=2
Sep 16 18:59:48.259431 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:48.259297    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" event={"ID":"9b635504-47bb-4126-aed6-53ac7014c69a","Type":"ContainerDied","Data":"9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae"}
Sep 16 18:59:49.465477 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:49.465436    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 16 18:59:49.470753 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:49.470725    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 18:59:52.527573 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.527507    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:59:52.628611 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.628575    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kwfb\" (UniqueName: \"kubernetes.io/projected/9b635504-47bb-4126-aed6-53ac7014c69a-kube-api-access-9kwfb\") pod \"9b635504-47bb-4126-aed6-53ac7014c69a\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") "
Sep 16 18:59:52.628847 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.628627    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-kube-rbac-proxy-sar-config\") pod \"9b635504-47bb-4126-aed6-53ac7014c69a\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") "
Sep 16 18:59:52.628847 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.628687    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-cabundle-cert\") pod \"9b635504-47bb-4126-aed6-53ac7014c69a\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") "
Sep 16 18:59:52.628847 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.628725    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b635504-47bb-4126-aed6-53ac7014c69a-kserve-provision-location\") pod \"9b635504-47bb-4126-aed6-53ac7014c69a\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") "
Sep 16 18:59:52.628847 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.628759    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b635504-47bb-4126-aed6-53ac7014c69a-proxy-tls\") pod \"9b635504-47bb-4126-aed6-53ac7014c69a\" (UID: \"9b635504-47bb-4126-aed6-53ac7014c69a\") "
Sep 16 18:59:52.629053 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.629031    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b635504-47bb-4126-aed6-53ac7014c69a" (UID: "9b635504-47bb-4126-aed6-53ac7014c69a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:52.629115 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.629073    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b635504-47bb-4126-aed6-53ac7014c69a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b635504-47bb-4126-aed6-53ac7014c69a" (UID: "9b635504-47bb-4126-aed6-53ac7014c69a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:52.629115 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.629108    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b635504-47bb-4126-aed6-53ac7014c69a" (UID: "9b635504-47bb-4126-aed6-53ac7014c69a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:52.631003 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.630980    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b635504-47bb-4126-aed6-53ac7014c69a-kube-api-access-9kwfb" (OuterVolumeSpecName: "kube-api-access-9kwfb") pod "9b635504-47bb-4126-aed6-53ac7014c69a" (UID: "9b635504-47bb-4126-aed6-53ac7014c69a"). InnerVolumeSpecName "kube-api-access-9kwfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:52.631091 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.631008    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b635504-47bb-4126-aed6-53ac7014c69a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b635504-47bb-4126-aed6-53ac7014c69a" (UID: "9b635504-47bb-4126-aed6-53ac7014c69a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:52.729990 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.729950    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-cabundle-cert\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 18:59:52.729990 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.729984    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b635504-47bb-4126-aed6-53ac7014c69a-kserve-provision-location\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 18:59:52.729990 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.729999    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b635504-47bb-4126-aed6-53ac7014c69a-proxy-tls\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 18:59:52.730229 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.730011    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kwfb\" (UniqueName: \"kubernetes.io/projected/9b635504-47bb-4126-aed6-53ac7014c69a-kube-api-access-9kwfb\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 18:59:52.730229 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:52.730022    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b635504-47bb-4126-aed6-53ac7014c69a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 18:59:53.285323 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.285282    2572 generic.go:356] "Generic (PLEG): container finished" podID="9b635504-47bb-4126-aed6-53ac7014c69a" containerID="f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d" exitCode=0
Sep 16 18:59:53.285492 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.285357    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"
Sep 16 18:59:53.285492 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.285348    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" event={"ID":"9b635504-47bb-4126-aed6-53ac7014c69a","Type":"ContainerDied","Data":"f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d"}
Sep 16 18:59:53.285492 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.285486    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv" event={"ID":"9b635504-47bb-4126-aed6-53ac7014c69a","Type":"ContainerDied","Data":"612a7550cbec961e96a309a14ccbda5e10cd9abadb5eaa0d0c3903c1fde1a3a1"}
Sep 16 18:59:53.285633 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.285527    2572 scope.go:117] "RemoveContainer" containerID="9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae"
Sep 16 18:59:53.295071 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.295054    2572 scope.go:117] "RemoveContainer" containerID="f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d"
Sep 16 18:59:53.303001 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.302982    2572 scope.go:117] "RemoveContainer" containerID="3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56"
Sep 16 18:59:53.308573 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.308550    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"]
Sep 16 18:59:53.311686 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.311657    2572 scope.go:117] "RemoveContainer" containerID="9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae"
Sep 16 18:59:53.312046 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:59:53.312021    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae\": container with ID starting with 9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae not found: ID does not exist" containerID="9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae"
Sep 16 18:59:53.312114 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.312058    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae"} err="failed to get container status \"9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae\": rpc error: code = NotFound desc = could not find container \"9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae\": container with ID starting with 9ff2ee4aab50b32ebc1d3c281e57a2087a950ec1f50e36c88ae5399d31aa0aae not found: ID does not exist"
Sep 16 18:59:53.312114 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.312082    2572 scope.go:117] "RemoveContainer" containerID="f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d"
Sep 16 18:59:53.312360 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:59:53.312343    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d\": container with ID starting with f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d not found: ID does not exist" containerID="f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d"
Sep 16 18:59:53.312410 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.312366    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d"} err="failed to get container status \"f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d\": rpc error: code = NotFound desc = could not find container \"f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d\": container with ID starting with f25d8c2c70e87e55e6763fe02e6667faff4ffbc9f1d6c72483f5e5a31625d54d not found: ID does not exist"
Sep 16 18:59:53.312410 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.312383    2572 scope.go:117] "RemoveContainer" containerID="3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56"
Sep 16 18:59:53.312585 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.312566    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-ch7fv"]
Sep 16 18:59:53.312682 ip-10-0-128-222 kubenswrapper[2572]: E0916 18:59:53.312667    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56\": container with ID starting with 3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56 not found: ID does not exist" containerID="3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56"
Sep 16 18:59:53.312763 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:53.312687    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56"} err="failed to get container status \"3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56\": rpc error: code = NotFound desc = could not find container \"3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56\": container with ID starting with 3a15c54f16d070c218251e874f01664697ee550aac89392e048f2ecc19cc9e56 not found: ID does not exist"
Sep 16 18:59:54.447726 ip-10-0-128-222 kubenswrapper[2572]: I0916 18:59:54.447697    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" path="/var/lib/kubelet/pods/9b635504-47bb-4126-aed6-53ac7014c69a/volumes"
Sep 16 19:00:02.786557 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.786502    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"]
Sep 16 19:00:02.786932 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.786844    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="storage-initializer"
Sep 16 19:00:02.786932 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.786855    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="storage-initializer"
Sep 16 19:00:02.786932 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.786864    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container"
Sep 16 19:00:02.786932 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.786869    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container"
Sep 16 19:00:02.786932 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.786881    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kube-rbac-proxy"
Sep 16 19:00:02.786932 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.786886    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kube-rbac-proxy"
Sep 16 19:00:02.787112 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.786937    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kube-rbac-proxy"
Sep 16 19:00:02.787112 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.786947    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b635504-47bb-4126-aed6-53ac7014c69a" containerName="kserve-container"
Sep 16 19:00:02.791183 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.791167    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:02.793316 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.793294    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-f7202-kube-rbac-proxy-sar-config\""
Sep 16 19:00:02.793404 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.793303    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-f7202-serving-cert\""
Sep 16 19:00:02.803598 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.803347    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"]
Sep 16 19:00:02.914057 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.914019    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-openshift-service-ca-bundle\") pod \"model-chainer-raw-f7202-69cb46ffc9-cdhvb\" (UID: \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\") " pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:02.914246 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:02.914089    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-proxy-tls\") pod \"model-chainer-raw-f7202-69cb46ffc9-cdhvb\" (UID: \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\") " pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:03.015058 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:03.015007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-openshift-service-ca-bundle\") pod \"model-chainer-raw-f7202-69cb46ffc9-cdhvb\" (UID: \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\") " pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:03.015248 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:03.015082    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-proxy-tls\") pod \"model-chainer-raw-f7202-69cb46ffc9-cdhvb\" (UID: \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\") " pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:03.015248 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:00:03.015190    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-f7202-serving-cert: secret "model-chainer-raw-f7202-serving-cert" not found
Sep 16 19:00:03.015362 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:00:03.015287    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-proxy-tls podName:7793990c-9ca6-45d3-a8f7-dbdf4b7517a7 nodeName:}" failed. No retries permitted until 2026-09-16 19:00:03.515264221 +0000 UTC m=+877.586361422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-proxy-tls") pod "model-chainer-raw-f7202-69cb46ffc9-cdhvb" (UID: "7793990c-9ca6-45d3-a8f7-dbdf4b7517a7") : secret "model-chainer-raw-f7202-serving-cert" not found
Sep 16 19:00:03.015705 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:03.015684    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-openshift-service-ca-bundle\") pod \"model-chainer-raw-f7202-69cb46ffc9-cdhvb\" (UID: \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\") " pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:03.519782 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:03.519737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-proxy-tls\") pod \"model-chainer-raw-f7202-69cb46ffc9-cdhvb\" (UID: \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\") " pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:03.522322 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:03.522298    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-proxy-tls\") pod \"model-chainer-raw-f7202-69cb46ffc9-cdhvb\" (UID: \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\") " pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:03.702679 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:03.702637    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:03.976680 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:03.976652    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"]
Sep 16 19:00:03.979855 ip-10-0-128-222 kubenswrapper[2572]: W0916 19:00:03.979829    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7793990c_9ca6_45d3_a8f7_dbdf4b7517a7.slice/crio-43eae6cddfa76a0cba86ebdffd0ef2d32bfe624b280bd6c5483df0e9688cfe2c WatchSource:0}: Error finding container 43eae6cddfa76a0cba86ebdffd0ef2d32bfe624b280bd6c5483df0e9688cfe2c: Status 404 returned error can't find the container with id 43eae6cddfa76a0cba86ebdffd0ef2d32bfe624b280bd6c5483df0e9688cfe2c
Sep 16 19:00:04.344597 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:04.344492    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" event={"ID":"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7","Type":"ContainerStarted","Data":"43eae6cddfa76a0cba86ebdffd0ef2d32bfe624b280bd6c5483df0e9688cfe2c"}
Sep 16 19:00:06.354972 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:06.354877    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" event={"ID":"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7","Type":"ContainerStarted","Data":"05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c"}
Sep 16 19:00:06.355341 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:06.354998    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:06.377983 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:06.377932    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" podStartSLOduration=2.272872639 podStartE2EDuration="4.377913802s" totalImagesPullingTime="2.105041163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:06.376555259 +0000 UTC m=+880.447652483" watchObservedRunningTime="2026-09-16 19:00:06.377913802 +0000 UTC m=+880.449011065"
Sep 16 19:00:12.365582 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:12.365553    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:12.863337 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:12.863301    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"]
Sep 16 19:00:12.863618 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:12.863592    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202" containerID="cri-o://05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c" gracePeriod=30
Sep 16 19:00:12.995598 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:12.995564    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"]
Sep 16 19:00:12.995834 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:12.995813    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" containerID="cri-o://927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77" gracePeriod=30
Sep 16 19:00:12.995905 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:12.995858    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kube-rbac-proxy" containerID="cri-o://bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558" gracePeriod=30
Sep 16 19:00:13.391082 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:13.390996    2572 generic.go:356] "Generic (PLEG): container finished" podID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerID="bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558" exitCode=2
Sep 16 19:00:13.391082 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:13.391048    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" event={"ID":"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd","Type":"ContainerDied","Data":"bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558"}
Sep 16 19:00:15.900524 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:15.900481    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 16 19:00:15.904969 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:15.904937    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 16 19:00:17.364066 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:17.364018    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:17.848754 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:17.848727    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 19:00:18.045852 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.045816    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-proxy-tls\") pod \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") "
Sep 16 19:00:18.046038 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.045895    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs5gj\" (UniqueName: \"kubernetes.io/projected/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-api-access-hs5gj\") pod \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") "
Sep 16 19:00:18.046038 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.045934    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kserve-provision-location\") pod \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") "
Sep 16 19:00:18.046038 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.045961    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-cabundle-cert\") pod \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") "
Sep 16 19:00:18.046038 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.045976    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-rbac-proxy-sar-config\") pod \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\" (UID: \"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd\") "
Sep 16 19:00:18.046277 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.046247    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" (UID: "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:18.046373 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.046351    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" (UID: "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:18.046478 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.046454    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" (UID: "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:18.048288 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.048265    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-api-access-hs5gj" (OuterVolumeSpecName: "kube-api-access-hs5gj") pod "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" (UID: "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd"). InnerVolumeSpecName "kube-api-access-hs5gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:18.048351 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.048270    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" (UID: "76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:18.146607 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.146575    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-proxy-tls\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:00:18.146607 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.146602    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5gj\" (UniqueName: \"kubernetes.io/projected/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-api-access-hs5gj\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:00:18.146607 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.146613    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kserve-provision-location\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:00:18.146838 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.146622    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-cabundle-cert\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:00:18.146838 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.146632    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:00:18.415048 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.414958    2572 generic.go:356] "Generic (PLEG): container finished" podID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerID="927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77" exitCode=0
Sep 16 19:00:18.415048 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.415008    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"
Sep 16 19:00:18.415605 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.415046    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" event={"ID":"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd","Type":"ContainerDied","Data":"927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77"}
Sep 16 19:00:18.415605 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.415124    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c" event={"ID":"76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd","Type":"ContainerDied","Data":"4545cbfd786153ac7e783c77663e0648398e2e0ee0bbc13e6e92f0ccb7bfe9b9"}
Sep 16 19:00:18.415605 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.415165    2572 scope.go:117] "RemoveContainer" containerID="bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558"
Sep 16 19:00:18.424731 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.424712    2572 scope.go:117] "RemoveContainer" containerID="927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77"
Sep 16 19:00:18.433408 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.433382    2572 scope.go:117] "RemoveContainer" containerID="73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d"
Sep 16 19:00:18.443360 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.443321    2572 scope.go:117] "RemoveContainer" containerID="bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558"
Sep 16 19:00:18.443788 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:00:18.443764    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558\": container with ID starting with bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558 not found: ID does not exist" containerID="bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558"
Sep 16 19:00:18.443932 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.443800    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558"} err="failed to get container status \"bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558\": rpc error: code = NotFound desc = could not find container \"bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558\": container with ID starting with bf4d851d3527d4aec9d8d5556463d7f4cb8c5b6e2723d1160d61f2a82b53d558 not found: ID does not exist"
Sep 16 19:00:18.443932 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.443821    2572 scope.go:117] "RemoveContainer" containerID="927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77"
Sep 16 19:00:18.444757 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:00:18.444731    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77\": container with ID starting with 927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77 not found: ID does not exist" containerID="927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77"
Sep 16 19:00:18.444931 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.444890    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77"} err="failed to get container status \"927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77\": rpc error: code = NotFound desc = could not find container \"927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77\": container with ID starting with 927d3faba9f38775f03ed9d99513273935eee7785dec34bc9d8e7cd8fe9e2d77 not found: ID does not exist"
Sep 16 19:00:18.444931 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.444927    2572 scope.go:117] "RemoveContainer" containerID="73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d"
Sep 16 19:00:18.445314 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:00:18.445284    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d\": container with ID starting with 73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d not found: ID does not exist" containerID="73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d"
Sep 16 19:00:18.445390 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.445324    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d"} err="failed to get container status \"73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d\": rpc error: code = NotFound desc = could not find container \"73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d\": container with ID starting with 73f3e88174509b814b6df2af9178e2c761de662b485ae1458d84dbed2dc22e6d not found: ID does not exist"
Sep 16 19:00:18.448586 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.448565    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"]
Sep 16 19:00:18.448686 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:18.448599    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f7202-predictor-55bcf87df4-rs22c"]
Sep 16 19:00:20.453397 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:20.453360    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" path="/var/lib/kubelet/pods/76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd/volumes"
Sep 16 19:00:22.363395 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:22.363358    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:27.363194 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:27.363149    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:27.363591 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:27.363296    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:32.363960 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:32.363920    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:37.363335 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:37.363290    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:42.363821 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:42.363781    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:43.521184 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.521156    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:43.526440 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.526415    2572 generic.go:356] "Generic (PLEG): container finished" podID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerID="05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c" exitCode=0
Sep 16 19:00:43.526610 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.526458    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" event={"ID":"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7","Type":"ContainerDied","Data":"05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c"}
Sep 16 19:00:43.526610 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.526461    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"
Sep 16 19:00:43.526610 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.526500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb" event={"ID":"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7","Type":"ContainerDied","Data":"43eae6cddfa76a0cba86ebdffd0ef2d32bfe624b280bd6c5483df0e9688cfe2c"}
Sep 16 19:00:43.526610 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.526548    2572 scope.go:117] "RemoveContainer" containerID="05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c"
Sep 16 19:00:43.534598 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.534579    2572 scope.go:117] "RemoveContainer" containerID="05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c"
Sep 16 19:00:43.534911 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:00:43.534891    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c\": container with ID starting with 05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c not found: ID does not exist" containerID="05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c"
Sep 16 19:00:43.534979 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.534920    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c"} err="failed to get container status \"05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c\": rpc error: code = NotFound desc = could not find container \"05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c\": container with ID starting with 05700cd902348e7d46912d1049bcc336ec514bc046f06901bd96c5744496446c not found: ID does not exist"
Sep 16 19:00:43.636733 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.636637    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-proxy-tls\") pod \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\" (UID: \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\") "
Sep 16 19:00:43.636733 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.636701    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-openshift-service-ca-bundle\") pod \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\" (UID: \"7793990c-9ca6-45d3-a8f7-dbdf4b7517a7\") "
Sep 16 19:00:43.637092 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.637066    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" (UID: "7793990c-9ca6-45d3-a8f7-dbdf4b7517a7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:43.638938 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.638917    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" (UID: "7793990c-9ca6-45d3-a8f7-dbdf4b7517a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:43.737390 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.737355    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-proxy-tls\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:00:43.737390 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.737384    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7-openshift-service-ca-bundle\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:00:43.847449 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.847416    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"]
Sep 16 19:00:43.858897 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:43.858866    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-f7202-69cb46ffc9-cdhvb"]
Sep 16 19:00:44.448073 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:00:44.448035    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" path="/var/lib/kubelet/pods/7793990c-9ca6-45d3-a8f7-dbdf4b7517a7/volumes"
Sep 16 19:02:03.134288 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134211    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"]
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134560    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="storage-initializer"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134576    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="storage-initializer"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134605    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134612    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134621    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134626    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134634    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kube-rbac-proxy"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134639    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kube-rbac-proxy"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134700    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kube-rbac-proxy"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134709    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7793990c-9ca6-45d3-a8f7-dbdf4b7517a7" containerName="model-chainer-raw-f7202"
Sep 16 19:02:03.134761 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.134717    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d94e8a-c5bf-4f5a-b2a6-a2b91fc88dbd" containerName="kserve-container"
Sep 16 19:02:03.137612 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.137594    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:03.139952 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.139928    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-9ebe2-serving-cert\""
Sep 16 19:02:03.140084 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.140041    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:02:03.140182 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.140080    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-9ebe2-kube-rbac-proxy-sar-config\""
Sep 16 19:02:03.140254 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.140213    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 19:02:03.149281 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.149261    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"]
Sep 16 19:02:03.190085 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.190045    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-proxy-tls\") pod \"model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf\" (UID: \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:03.190276 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.190104    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf\" (UID: \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:03.291422 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.291376    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-proxy-tls\") pod \"model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf\" (UID: \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:03.291645 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.291449    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf\" (UID: \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:03.292117 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.292097    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf\" (UID: \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:03.294050 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.294018    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-proxy-tls\") pod \"model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf\" (UID: \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:03.447709 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.447676    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:03.722835 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.722752    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"]
Sep 16 19:02:03.888443 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.888407    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" event={"ID":"d07b067d-5b13-4fa1-98f7-d95aeb40f11d","Type":"ContainerStarted","Data":"08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d"}
Sep 16 19:02:03.888652 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.888457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" event={"ID":"d07b067d-5b13-4fa1-98f7-d95aeb40f11d","Type":"ContainerStarted","Data":"9ba14f897dcedee058d89b69970129273160512767eeeeb4a2e8343186bef249"}
Sep 16 19:02:03.888652 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.888571    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:03.926372 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:03.926308    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" podStartSLOduration=0.926287478 podStartE2EDuration="926.287478ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:03.92579147 +0000 UTC m=+997.996888697" watchObservedRunningTime="2026-09-16 19:02:03.926287478 +0000 UTC m=+997.997384703"
Sep 16 19:02:09.898636 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:09.898608    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:13.204061 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:13.204027    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"]
Sep 16 19:02:13.204491 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:13.204255    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2" containerID="cri-o://08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d" gracePeriod=30
Sep 16 19:02:14.896739 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:14.896698    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:02:19.896998 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:19.896959    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:02:24.897202 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:24.897164    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:02:24.897596 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:24.897271    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:29.896305 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:29.896258    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:02:34.897176 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:34.897139    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:02:39.896899 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:39.896854    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:02:43.355199 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:43.355175    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:43.426668 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:43.426639    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-proxy-tls\") pod \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\" (UID: \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\") "
Sep 16 19:02:43.426854 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:43.426675    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-openshift-service-ca-bundle\") pod \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\" (UID: \"d07b067d-5b13-4fa1-98f7-d95aeb40f11d\") "
Sep 16 19:02:43.427064 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:43.427042    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d07b067d-5b13-4fa1-98f7-d95aeb40f11d" (UID: "d07b067d-5b13-4fa1-98f7-d95aeb40f11d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:43.428872 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:43.428852    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d07b067d-5b13-4fa1-98f7-d95aeb40f11d" (UID: "d07b067d-5b13-4fa1-98f7-d95aeb40f11d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:43.527719 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:43.527638    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-proxy-tls\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:02:43.527719 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:43.527667    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d07b067d-5b13-4fa1-98f7-d95aeb40f11d-openshift-service-ca-bundle\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:02:44.064813 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.064784    2572 generic.go:356] "Generic (PLEG): container finished" podID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerID="08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d" exitCode=0
Sep 16 19:02:44.064988 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.064826    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"
Sep 16 19:02:44.064988 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.064875    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" event={"ID":"d07b067d-5b13-4fa1-98f7-d95aeb40f11d","Type":"ContainerDied","Data":"08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d"}
Sep 16 19:02:44.064988 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.064932    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf" event={"ID":"d07b067d-5b13-4fa1-98f7-d95aeb40f11d","Type":"ContainerDied","Data":"9ba14f897dcedee058d89b69970129273160512767eeeeb4a2e8343186bef249"}
Sep 16 19:02:44.064988 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.064967    2572 scope.go:117] "RemoveContainer" containerID="08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d"
Sep 16 19:02:44.074050 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.074035    2572 scope.go:117] "RemoveContainer" containerID="08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d"
Sep 16 19:02:44.074315 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:02:44.074296    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d\": container with ID starting with 08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d not found: ID does not exist" containerID="08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d"
Sep 16 19:02:44.074360 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.074325    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d"} err="failed to get container status \"08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d\": rpc error: code = NotFound desc = could not find container \"08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d\": container with ID starting with 08a1fcce06d9835979fd0a58cc52b7a83014c1f97eee564384d993a4ae10a53d not found: ID does not exist"
Sep 16 19:02:44.094314 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.094295    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"]
Sep 16 19:02:44.100991 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.100971    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-9ebe2-7fcdd4444c-bhswf"]
Sep 16 19:02:44.446614 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:02:44.446588    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" path="/var/lib/kubelet/pods/d07b067d-5b13-4fa1-98f7-d95aeb40f11d/volumes"
Sep 16 19:03:51.722095 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.722057    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"]
Sep 16 19:03:51.722563 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.722374    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2"
Sep 16 19:03:51.722563 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.722385    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2"
Sep 16 19:03:51.722563 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.722450    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d07b067d-5b13-4fa1-98f7-d95aeb40f11d" containerName="model-chainer-raw-hpa-9ebe2"
Sep 16 19:03:51.726727 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.726709    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.729030 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.728999    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:51.729158 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.729105    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-582b75-dockercfg-4j664\""
Sep 16 19:03:51.729216 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.728999    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-582b75-kube-rbac-proxy-sar-config\""
Sep 16 19:03:51.729262 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.729217    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-582b75-predictor-serving-cert\""
Sep 16 19:03:51.729765 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.729738    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:51.729765 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.729761    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-582b75\""
Sep 16 19:03:51.729904 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.729780    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:51.736853 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.736832    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"]
Sep 16 19:03:51.767605 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.767570    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13134712-6e89-41a9-8f8e-26fd99a92c96-kserve-provision-location\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.767605 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.767605    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-cabundle-cert\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.767794 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.767640    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptbjl\" (UniqueName: \"kubernetes.io/projected/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-api-access-ptbjl\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.767794 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.767708    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.767794 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.767759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13134712-6e89-41a9-8f8e-26fd99a92c96-proxy-tls\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.868932 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.868887    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.869112 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.868948    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13134712-6e89-41a9-8f8e-26fd99a92c96-proxy-tls\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.869112 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.868997    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13134712-6e89-41a9-8f8e-26fd99a92c96-kserve-provision-location\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.869112 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.869021    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-cabundle-cert\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.869112 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.869074    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptbjl\" (UniqueName: \"kubernetes.io/projected/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-api-access-ptbjl\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.869473 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.869449    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13134712-6e89-41a9-8f8e-26fd99a92c96-kserve-provision-location\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.869690 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.869673    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-cabundle-cert\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.869861 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.869675    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.871622 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.871602    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13134712-6e89-41a9-8f8e-26fd99a92c96-proxy-tls\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:51.877977 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:51.877957    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptbjl\" (UniqueName: \"kubernetes.io/projected/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-api-access-ptbjl\") pod \"isvc-init-fail-582b75-predictor-76fb588c48-467v7\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") " pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:52.037957 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:52.037847    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:03:52.313934 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:52.313908    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"]
Sep 16 19:03:52.315879 ip-10-0-128-222 kubenswrapper[2572]: W0916 19:03:52.315852    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13134712_6e89_41a9_8f8e_26fd99a92c96.slice/crio-6790a60d0aad74dd36cafe6ac3d39236f9204cd0fd4e16ac853aa3da9dde5ddf WatchSource:0}: Error finding container 6790a60d0aad74dd36cafe6ac3d39236f9204cd0fd4e16ac853aa3da9dde5ddf: Status 404 returned error can't find the container with id 6790a60d0aad74dd36cafe6ac3d39236f9204cd0fd4e16ac853aa3da9dde5ddf
Sep 16 19:03:52.370040 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:52.370011    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7" event={"ID":"13134712-6e89-41a9-8f8e-26fd99a92c96","Type":"ContainerStarted","Data":"6790a60d0aad74dd36cafe6ac3d39236f9204cd0fd4e16ac853aa3da9dde5ddf"}
Sep 16 19:03:53.376816 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:53.376782    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7" event={"ID":"13134712-6e89-41a9-8f8e-26fd99a92c96","Type":"ContainerStarted","Data":"994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f"}
Sep 16 19:03:55.386288 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:55.386259    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-582b75-predictor-76fb588c48-467v7_13134712-6e89-41a9-8f8e-26fd99a92c96/storage-initializer/0.log"
Sep 16 19:03:55.386681 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:55.386299    2572 generic.go:356] "Generic (PLEG): container finished" podID="13134712-6e89-41a9-8f8e-26fd99a92c96" containerID="994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f" exitCode=1
Sep 16 19:03:55.386681 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:55.386378    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7" event={"ID":"13134712-6e89-41a9-8f8e-26fd99a92c96","Type":"ContainerDied","Data":"994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f"}
Sep 16 19:03:56.393052 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:56.393022    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-582b75-predictor-76fb588c48-467v7_13134712-6e89-41a9-8f8e-26fd99a92c96/storage-initializer/0.log"
Sep 16 19:03:56.393558 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:56.393148    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7" event={"ID":"13134712-6e89-41a9-8f8e-26fd99a92c96","Type":"ContainerStarted","Data":"ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7"}
Sep 16 19:03:56.684471 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:56.684394    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"]
Sep 16 19:03:57.397601 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:03:57.397561    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7" podUID="13134712-6e89-41a9-8f8e-26fd99a92c96" containerName="storage-initializer" containerID="cri-o://ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7" gracePeriod=30
Sep 16 19:04:02.242947 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.242926    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-582b75-predictor-76fb588c48-467v7_13134712-6e89-41a9-8f8e-26fd99a92c96/storage-initializer/1.log"
Sep 16 19:04:02.243336 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.243317    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-582b75-predictor-76fb588c48-467v7_13134712-6e89-41a9-8f8e-26fd99a92c96/storage-initializer/0.log"
Sep 16 19:04:02.245809 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.245794    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:04:02.356820 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.356735    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13134712-6e89-41a9-8f8e-26fd99a92c96-kserve-provision-location\") pod \"13134712-6e89-41a9-8f8e-26fd99a92c96\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") "
Sep 16 19:04:02.356975 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.356830    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13134712-6e89-41a9-8f8e-26fd99a92c96-proxy-tls\") pod \"13134712-6e89-41a9-8f8e-26fd99a92c96\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") "
Sep 16 19:04:02.356975 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.356861    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-cabundle-cert\") pod \"13134712-6e89-41a9-8f8e-26fd99a92c96\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") "
Sep 16 19:04:02.356975 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.356880    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-rbac-proxy-sar-config\") pod \"13134712-6e89-41a9-8f8e-26fd99a92c96\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") "
Sep 16 19:04:02.356975 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.356924    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptbjl\" (UniqueName: \"kubernetes.io/projected/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-api-access-ptbjl\") pod \"13134712-6e89-41a9-8f8e-26fd99a92c96\" (UID: \"13134712-6e89-41a9-8f8e-26fd99a92c96\") "
Sep 16 19:04:02.357112 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.357054    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13134712-6e89-41a9-8f8e-26fd99a92c96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13134712-6e89-41a9-8f8e-26fd99a92c96" (UID: "13134712-6e89-41a9-8f8e-26fd99a92c96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:02.357206 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.357190    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13134712-6e89-41a9-8f8e-26fd99a92c96-kserve-provision-location\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:04:02.357256 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.357239    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13134712-6e89-41a9-8f8e-26fd99a92c96" (UID: "13134712-6e89-41a9-8f8e-26fd99a92c96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:02.357315 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.357264    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13134712-6e89-41a9-8f8e-26fd99a92c96" (UID: "13134712-6e89-41a9-8f8e-26fd99a92c96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:02.359209 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.359177    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-api-access-ptbjl" (OuterVolumeSpecName: "kube-api-access-ptbjl") pod "13134712-6e89-41a9-8f8e-26fd99a92c96" (UID: "13134712-6e89-41a9-8f8e-26fd99a92c96"). InnerVolumeSpecName "kube-api-access-ptbjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:02.359322 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.359232    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13134712-6e89-41a9-8f8e-26fd99a92c96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13134712-6e89-41a9-8f8e-26fd99a92c96" (UID: "13134712-6e89-41a9-8f8e-26fd99a92c96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:02.420236 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.420208    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-582b75-predictor-76fb588c48-467v7_13134712-6e89-41a9-8f8e-26fd99a92c96/storage-initializer/1.log"
Sep 16 19:04:02.420666 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.420651    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-582b75-predictor-76fb588c48-467v7_13134712-6e89-41a9-8f8e-26fd99a92c96/storage-initializer/0.log"
Sep 16 19:04:02.420723 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.420686    2572 generic.go:356] "Generic (PLEG): container finished" podID="13134712-6e89-41a9-8f8e-26fd99a92c96" containerID="ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7" exitCode=1
Sep 16 19:04:02.420760 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.420733    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7" event={"ID":"13134712-6e89-41a9-8f8e-26fd99a92c96","Type":"ContainerDied","Data":"ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7"}
Sep 16 19:04:02.420760 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.420745    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"
Sep 16 19:04:02.420825 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.420810    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7" event={"ID":"13134712-6e89-41a9-8f8e-26fd99a92c96","Type":"ContainerDied","Data":"6790a60d0aad74dd36cafe6ac3d39236f9204cd0fd4e16ac853aa3da9dde5ddf"}
Sep 16 19:04:02.420858 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.420837    2572 scope.go:117] "RemoveContainer" containerID="ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7"
Sep 16 19:04:02.430208 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.430191    2572 scope.go:117] "RemoveContainer" containerID="994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f"
Sep 16 19:04:02.437728 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.437709    2572 scope.go:117] "RemoveContainer" containerID="ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7"
Sep 16 19:04:02.437998 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:04:02.437980    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7\": container with ID starting with ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7 not found: ID does not exist" containerID="ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7"
Sep 16 19:04:02.438062 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.438009    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7"} err="failed to get container status \"ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7\": rpc error: code = NotFound desc = could not find container \"ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7\": container with ID starting with ce0dcc3b4ab86cf320b4d9e5848766627bb2c1be4fd14ca8ab7c08efe64bffb7 not found: ID does not exist"
Sep 16 19:04:02.438062 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.438035    2572 scope.go:117] "RemoveContainer" containerID="994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f"
Sep 16 19:04:02.438301 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:04:02.438285    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f\": container with ID starting with 994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f not found: ID does not exist" containerID="994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f"
Sep 16 19:04:02.438340 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.438307    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f"} err="failed to get container status \"994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f\": rpc error: code = NotFound desc = could not find container \"994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f\": container with ID starting with 994de308f5616675feffd4bcd3c0e7d0c01fa193020b75b62b775d80c88dae0f not found: ID does not exist"
Sep 16 19:04:02.457645 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.457617    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13134712-6e89-41a9-8f8e-26fd99a92c96-proxy-tls\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:04:02.457645 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.457642    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-cabundle-cert\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:04:02.457790 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.457651    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:04:02.457790 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.457660    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptbjl\" (UniqueName: \"kubernetes.io/projected/13134712-6e89-41a9-8f8e-26fd99a92c96-kube-api-access-ptbjl\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:04:02.463485 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.463459    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"]
Sep 16 19:04:02.471644 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:02.471623    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-582b75-predictor-76fb588c48-467v7"]
Sep 16 19:04:04.447615 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:04.447574    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13134712-6e89-41a9-8f8e-26fd99a92c96" path="/var/lib/kubelet/pods/13134712-6e89-41a9-8f8e-26fd99a92c96/volumes"
Sep 16 19:04:08.753309 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.753270    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"]
Sep 16 19:04:08.753710 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.753667    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13134712-6e89-41a9-8f8e-26fd99a92c96" containerName="storage-initializer"
Sep 16 19:04:08.753710 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.753680    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="13134712-6e89-41a9-8f8e-26fd99a92c96" containerName="storage-initializer"
Sep 16 19:04:08.753780 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.753735    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="13134712-6e89-41a9-8f8e-26fd99a92c96" containerName="storage-initializer"
Sep 16 19:04:08.753780 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.753743    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="13134712-6e89-41a9-8f8e-26fd99a92c96" containerName="storage-initializer"
Sep 16 19:04:08.753840 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.753799    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13134712-6e89-41a9-8f8e-26fd99a92c96" containerName="storage-initializer"
Sep 16 19:04:08.753840 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.753806    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="13134712-6e89-41a9-8f8e-26fd99a92c96" containerName="storage-initializer"
Sep 16 19:04:08.759123 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.759101    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.761557 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.761495    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:04:08.761688 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.761580    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-1ed38-kube-rbac-proxy-sar-config\""
Sep 16 19:04:08.761688 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.761599    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:04:08.761688 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.761559    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:04:08.761688 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.761565    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 19:04:08.761924 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.761561    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:04:08.761989 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.761977    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-1ed38-predictor-serving-cert\""
Sep 16 19:04:08.769213 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.769187    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"]
Sep 16 19:04:08.812137 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.812101    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbx6c\" (UniqueName: \"kubernetes.io/projected/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-api-access-bbx6c\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.812301 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.812150    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-proxy-tls\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.812301 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.812178    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.812301 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.812232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.812404 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.812304    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.913626 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.913585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.913821 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.913640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbx6c\" (UniqueName: \"kubernetes.io/projected/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-api-access-bbx6c\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.913821 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.913669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-proxy-tls\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.913821 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.913697    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.913821 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.913714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.914177 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.914151    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.914355 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.914335    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.914411 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.914360    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.916316 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.916287    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-proxy-tls\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:08.928390 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:08.928360    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbx6c\" (UniqueName: \"kubernetes.io/projected/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-api-access-bbx6c\") pod \"isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:09.070102 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:09.069993    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:09.344864 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:09.344756    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"]
Sep 16 19:04:09.347197 ip-10-0-128-222 kubenswrapper[2572]: W0916 19:04:09.347169    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f9d2db_6097_4ea6_9de7_70304a5ee0a5.slice/crio-c0cc548d5bd92ab2c9340e2aa62a30de23ca141e1638de343f73ac1a7a57142c WatchSource:0}: Error finding container c0cc548d5bd92ab2c9340e2aa62a30de23ca141e1638de343f73ac1a7a57142c: Status 404 returned error can't find the container with id c0cc548d5bd92ab2c9340e2aa62a30de23ca141e1638de343f73ac1a7a57142c
Sep 16 19:04:09.453386 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:09.453351    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" event={"ID":"41f9d2db-6097-4ea6-9de7-70304a5ee0a5","Type":"ContainerStarted","Data":"ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463"}
Sep 16 19:04:09.453582 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:09.453409    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" event={"ID":"41f9d2db-6097-4ea6-9de7-70304a5ee0a5","Type":"ContainerStarted","Data":"c0cc548d5bd92ab2c9340e2aa62a30de23ca141e1638de343f73ac1a7a57142c"}
Sep 16 19:04:13.472643 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:13.472551    2572 generic.go:356] "Generic (PLEG): container finished" podID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerID="ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463" exitCode=0
Sep 16 19:04:13.472643 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:13.472626    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" event={"ID":"41f9d2db-6097-4ea6-9de7-70304a5ee0a5","Type":"ContainerDied","Data":"ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463"}
Sep 16 19:04:14.479121 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:14.479086    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" event={"ID":"41f9d2db-6097-4ea6-9de7-70304a5ee0a5","Type":"ContainerStarted","Data":"90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723"}
Sep 16 19:04:14.479493 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:14.479150    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" event={"ID":"41f9d2db-6097-4ea6-9de7-70304a5ee0a5","Type":"ContainerStarted","Data":"afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb"}
Sep 16 19:04:14.479493 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:14.479423    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:14.479652 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:14.479555    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:14.481083 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:14.481054    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:04:14.500888 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:14.500841    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podStartSLOduration=2.702086256 podStartE2EDuration="6.500823227s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.798736971s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:14.499011723 +0000 UTC m=+1128.570108973" watchObservedRunningTime="2026-09-16 19:04:14.500823227 +0000 UTC m=+1128.571920450"
Sep 16 19:04:15.483959 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:15.483921    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:04:20.489280 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:20.489250    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:04:20.489736 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:20.489709    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:04:30.490218 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:30.490179    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:04:40.490685 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:40.490646    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:04:50.490085 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:04:50.489998    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:05:00.490468 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:05:00.490426    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:05:10.490260 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:05:10.490215    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:05:20.489803 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:05:20.489765    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:05:30.490023 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:05:30.489977    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:05:40.489992 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:05:40.489949    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:05:45.443081 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:05:45.443035    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:05:55.442964 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:05:55.442915    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:06:05.443042 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:05.442994    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:06:15.443108 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:15.443064    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:06:25.443473 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:25.443364    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:06:35.443584 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:35.443537    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:06:45.443439 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:45.443392    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:06:55.444118 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:55.444087    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:06:58.874179 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:58.874147    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"]
Sep 16 19:06:58.874686 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:58.874422    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" containerID="cri-o://afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb" gracePeriod=30
Sep 16 19:06:58.874686 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:58.874450    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kube-rbac-proxy" containerID="cri-o://90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723" gracePeriod=30
Sep 16 19:06:59.215264 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:59.215230    2572 generic.go:356] "Generic (PLEG): container finished" podID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerID="90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723" exitCode=2
Sep 16 19:06:59.215450 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:06:59.215280    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" event={"ID":"41f9d2db-6097-4ea6-9de7-70304a5ee0a5","Type":"ContainerDied","Data":"90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723"}
Sep 16 19:07:00.485131 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:00.485081    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 19:07:04.235387 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.235353    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"]
Sep 16 19:07:04.239321 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.239305    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.242035 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.242013    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:07:04.242422 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.242404    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:07:04.259766 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.259736    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.259894 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.259780    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a3fb07-714c-43e9-b93b-e15732c4e467-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.259894 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.259819    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.259894 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.259879    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5gz6\" (UniqueName: \"kubernetes.io/projected/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-api-access-k5gz6\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.260024 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.259928    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a3fb07-714c-43e9-b93b-e15732c4e467-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.260408 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.260391    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"]
Sep 16 19:07:04.360853 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.360820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gz6\" (UniqueName: \"kubernetes.io/projected/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-api-access-k5gz6\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.361068 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.360860    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a3fb07-714c-43e9-b93b-e15732c4e467-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.361068 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.360900    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.361068 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.360926    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a3fb07-714c-43e9-b93b-e15732c4e467-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.361068 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.360954    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.361388 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.361364    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a3fb07-714c-43e9-b93b-e15732c4e467-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.361816 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.361794    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.361885 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.361794    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.363524 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.363495    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a3fb07-714c-43e9-b93b-e15732c4e467-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.369714 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.369694    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5gz6\" (UniqueName: \"kubernetes.io/projected/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-api-access-k5gz6\") pod \"isvc-canary-create-predictor-6fb974599-27mrk\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.549325 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.549217    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:04.831314 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:04.831286    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"]
Sep 16 19:07:04.833495 ip-10-0-128-222 kubenswrapper[2572]: W0916 19:07:04.833450    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a3fb07_714c_43e9_b93b_e15732c4e467.slice/crio-4d1eb9a8698466aac49ab46e8dbeb52b04e7fe903d88fa093d28f0df5bf6ebe8 WatchSource:0}: Error finding container 4d1eb9a8698466aac49ab46e8dbeb52b04e7fe903d88fa093d28f0df5bf6ebe8: Status 404 returned error can't find the container with id 4d1eb9a8698466aac49ab46e8dbeb52b04e7fe903d88fa093d28f0df5bf6ebe8
Sep 16 19:07:05.243320 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:05.243290    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" event={"ID":"31a3fb07-714c-43e9-b93b-e15732c4e467","Type":"ContainerStarted","Data":"86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5"}
Sep 16 19:07:05.243715 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:05.243343    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" event={"ID":"31a3fb07-714c-43e9-b93b-e15732c4e467","Type":"ContainerStarted","Data":"4d1eb9a8698466aac49ab46e8dbeb52b04e7fe903d88fa093d28f0df5bf6ebe8"}
Sep 16 19:07:05.443150 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:05.443107    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 16 19:07:05.485042 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:05.485009    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 16 19:07:09.226923 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.226894    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:07:09.263980 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.263880    2572 generic.go:356] "Generic (PLEG): container finished" podID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerID="afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb" exitCode=0
Sep 16 19:07:09.263980 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.263938    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"
Sep 16 19:07:09.263980 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.263949    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" event={"ID":"41f9d2db-6097-4ea6-9de7-70304a5ee0a5","Type":"ContainerDied","Data":"afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb"}
Sep 16 19:07:09.264240 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.264015    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2" event={"ID":"41f9d2db-6097-4ea6-9de7-70304a5ee0a5","Type":"ContainerDied","Data":"c0cc548d5bd92ab2c9340e2aa62a30de23ca141e1638de343f73ac1a7a57142c"}
Sep 16 19:07:09.264240 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.264053    2572 scope.go:117] "RemoveContainer" containerID="90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723"
Sep 16 19:07:09.265734 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.265675    2572 generic.go:356] "Generic (PLEG): container finished" podID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerID="86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5" exitCode=0
Sep 16 19:07:09.265734 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.265724    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" event={"ID":"31a3fb07-714c-43e9-b93b-e15732c4e467","Type":"ContainerDied","Data":"86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5"}
Sep 16 19:07:09.273113 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.273088    2572 scope.go:117] "RemoveContainer" containerID="afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb"
Sep 16 19:07:09.281440 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.281418    2572 scope.go:117] "RemoveContainer" containerID="ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463"
Sep 16 19:07:09.294115 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.294096    2572 scope.go:117] "RemoveContainer" containerID="90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723"
Sep 16 19:07:09.294454 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:07:09.294434    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723\": container with ID starting with 90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723 not found: ID does not exist" containerID="90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723"
Sep 16 19:07:09.294578 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.294462    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723"} err="failed to get container status \"90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723\": rpc error: code = NotFound desc = could not find container \"90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723\": container with ID starting with 90f58cc57df15f5962e86314b576d3e675b60175351ec1efca208183721d3723 not found: ID does not exist"
Sep 16 19:07:09.294578 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.294481    2572 scope.go:117] "RemoveContainer" containerID="afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb"
Sep 16 19:07:09.294818 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:07:09.294800    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb\": container with ID starting with afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb not found: ID does not exist" containerID="afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb"
Sep 16 19:07:09.294870 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.294825    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb"} err="failed to get container status \"afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb\": rpc error: code = NotFound desc = could not find container \"afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb\": container with ID starting with afeb8634a76313c1fe8804ea9caf2642e31b26ff414e7d5d060057ad395417cb not found: ID does not exist"
Sep 16 19:07:09.294870 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.294841    2572 scope.go:117] "RemoveContainer" containerID="ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463"
Sep 16 19:07:09.295089 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:07:09.295071    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463\": container with ID starting with ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463 not found: ID does not exist" containerID="ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463"
Sep 16 19:07:09.295127 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.295094    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463"} err="failed to get container status \"ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463\": rpc error: code = NotFound desc = could not find container \"ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463\": container with ID starting with ffcae08d76d2fcfa17c415834bbdd37fe26ae4b40c24ac90fd10a9e6541dc463 not found: ID does not exist"
Sep 16 19:07:09.300325 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.300304    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-cabundle-cert\") pod \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") "
Sep 16 19:07:09.300413 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.300367    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kserve-provision-location\") pod \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") "
Sep 16 19:07:09.300413 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.300410    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-proxy-tls\") pod \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") "
Sep 16 19:07:09.300481 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.300435    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-rbac-proxy-sar-config\") pod \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") "
Sep 16 19:07:09.300585 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.300478    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbx6c\" (UniqueName: \"kubernetes.io/projected/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-api-access-bbx6c\") pod \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\" (UID: \"41f9d2db-6097-4ea6-9de7-70304a5ee0a5\") "
Sep 16 19:07:09.300916 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.300732    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41f9d2db-6097-4ea6-9de7-70304a5ee0a5" (UID: "41f9d2db-6097-4ea6-9de7-70304a5ee0a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:07:09.300916 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.300784    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41f9d2db-6097-4ea6-9de7-70304a5ee0a5" (UID: "41f9d2db-6097-4ea6-9de7-70304a5ee0a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:09.300916 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.300812    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kserve-provision-location\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:07:09.300916 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.300853    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41f9d2db-6097-4ea6-9de7-70304a5ee0a5" (UID: "41f9d2db-6097-4ea6-9de7-70304a5ee0a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:07:09.302632 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.302607    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41f9d2db-6097-4ea6-9de7-70304a5ee0a5" (UID: "41f9d2db-6097-4ea6-9de7-70304a5ee0a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:09.302834 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.302808    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-api-access-bbx6c" (OuterVolumeSpecName: "kube-api-access-bbx6c") pod "41f9d2db-6097-4ea6-9de7-70304a5ee0a5" (UID: "41f9d2db-6097-4ea6-9de7-70304a5ee0a5"). InnerVolumeSpecName "kube-api-access-bbx6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:09.401869 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.401838    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-proxy-tls\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:07:09.401869 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.401866    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:07:09.401869 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.401876    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbx6c\" (UniqueName: \"kubernetes.io/projected/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-kube-api-access-bbx6c\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:07:09.402081 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.401884    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f9d2db-6097-4ea6-9de7-70304a5ee0a5-cabundle-cert\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:07:09.593337 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.593293    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"]
Sep 16 19:07:09.597218 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:09.597190    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1ed38-predictor-69ffcf968-5xhh2"]
Sep 16 19:07:10.273315 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:10.273281    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" event={"ID":"31a3fb07-714c-43e9-b93b-e15732c4e467","Type":"ContainerStarted","Data":"702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729"}
Sep 16 19:07:10.273704 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:10.273346    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" event={"ID":"31a3fb07-714c-43e9-b93b-e15732c4e467","Type":"ContainerStarted","Data":"f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a"}
Sep 16 19:07:10.273704 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:10.273548    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:10.298561 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:10.298485    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podStartSLOduration=2.624488269 podStartE2EDuration="6.298467599s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.67397933s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:10.296636975 +0000 UTC m=+1304.367734198" watchObservedRunningTime="2026-09-16 19:07:10.298467599 +0000 UTC m=+1304.369564880"
Sep 16 19:07:10.448207 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:10.448175    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" path="/var/lib/kubelet/pods/41f9d2db-6097-4ea6-9de7-70304a5ee0a5/volumes"
Sep 16 19:07:11.277740 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:11.277713    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:11.279067 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:11.279043    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 19:07:12.283570 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:12.283488    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 19:07:17.289380 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:17.289352    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:07:17.289871 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:17.289841    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 19:07:27.290024 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:27.289978    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 19:07:37.290825 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:37.290785    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 19:07:47.290738 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:47.290693    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 19:07:57.290809 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:07:57.290756    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 19:08:07.290122 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:07.290074    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 19:08:17.290218 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:17.290175    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 16 19:08:27.291327 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:27.291292    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:08:27.520958 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:27.520913    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"]
Sep 16 19:08:27.617423 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:27.617330    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container" containerID="cri-o://f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a" gracePeriod=30
Sep 16 19:08:27.617423 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:27.617351    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kube-rbac-proxy" containerID="cri-o://702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729" gracePeriod=30
Sep 16 19:08:28.624179 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:28.624147    2572 generic.go:356] "Generic (PLEG): container finished" podID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerID="702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729" exitCode=2
Sep 16 19:08:28.624575 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:28.624218    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" event={"ID":"31a3fb07-714c-43e9-b93b-e15732c4e467","Type":"ContainerDied","Data":"702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729"}
Sep 16 19:08:32.284673 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.284621    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 16 19:08:32.559754 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.559732    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:08:32.644388 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.644354    2572 generic.go:356] "Generic (PLEG): container finished" podID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerID="f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a" exitCode=0
Sep 16 19:08:32.644574 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.644406    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"
Sep 16 19:08:32.644574 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.644431    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" event={"ID":"31a3fb07-714c-43e9-b93b-e15732c4e467","Type":"ContainerDied","Data":"f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a"}
Sep 16 19:08:32.644574 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.644487    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk" event={"ID":"31a3fb07-714c-43e9-b93b-e15732c4e467","Type":"ContainerDied","Data":"4d1eb9a8698466aac49ab46e8dbeb52b04e7fe903d88fa093d28f0df5bf6ebe8"}
Sep 16 19:08:32.644713 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.644577    2572 scope.go:117] "RemoveContainer" containerID="702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729"
Sep 16 19:08:32.652764 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.652753    2572 scope.go:117] "RemoveContainer" containerID="f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a"
Sep 16 19:08:32.660131 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.660115    2572 scope.go:117] "RemoveContainer" containerID="86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5"
Sep 16 19:08:32.667247 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.667230    2572 scope.go:117] "RemoveContainer" containerID="702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729"
Sep 16 19:08:32.667502 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:08:32.667483    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729\": container with ID starting with 702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729 not found: ID does not exist" containerID="702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729"
Sep 16 19:08:32.667570 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.667528    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729"} err="failed to get container status \"702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729\": rpc error: code = NotFound desc = could not find container \"702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729\": container with ID starting with 702b37e62e2aef83b7b9583b9bea53c37682eb5b936c31ac532acdb4fdb64729 not found: ID does not exist"
Sep 16 19:08:32.667570 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.667549    2572 scope.go:117] "RemoveContainer" containerID="f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a"
Sep 16 19:08:32.667789 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:08:32.667770    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a\": container with ID starting with f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a not found: ID does not exist" containerID="f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a"
Sep 16 19:08:32.667854 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.667800    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a"} err="failed to get container status \"f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a\": rpc error: code = NotFound desc = could not find container \"f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a\": container with ID starting with f6576e173d61fec1e7030ddde1e023a9564e87c167b51595ca4958216960f42a not found: ID does not exist"
Sep 16 19:08:32.667854 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.667822    2572 scope.go:117] "RemoveContainer" containerID="86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5"
Sep 16 19:08:32.668089 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:08:32.668057    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5\": container with ID starting with 86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5 not found: ID does not exist" containerID="86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5"
Sep 16 19:08:32.668128 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.668094    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5"} err="failed to get container status \"86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5\": rpc error: code = NotFound desc = could not find container \"86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5\": container with ID starting with 86b217c2a6d7bfff7a4dd4becbd12bd210a2806102ee1fe97a5aae9790d4dbe5 not found: ID does not exist"
Sep 16 19:08:32.682892 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.682874    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a3fb07-714c-43e9-b93b-e15732c4e467-kserve-provision-location\") pod \"31a3fb07-714c-43e9-b93b-e15732c4e467\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") "
Sep 16 19:08:32.682971 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.682902    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-rbac-proxy-sar-config\") pod \"31a3fb07-714c-43e9-b93b-e15732c4e467\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") "
Sep 16 19:08:32.683039 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.682977    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-cabundle-cert\") pod \"31a3fb07-714c-43e9-b93b-e15732c4e467\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") "
Sep 16 19:08:32.683039 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.682998    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a3fb07-714c-43e9-b93b-e15732c4e467-proxy-tls\") pod \"31a3fb07-714c-43e9-b93b-e15732c4e467\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") "
Sep 16 19:08:32.683039 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.683023    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5gz6\" (UniqueName: \"kubernetes.io/projected/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-api-access-k5gz6\") pod \"31a3fb07-714c-43e9-b93b-e15732c4e467\" (UID: \"31a3fb07-714c-43e9-b93b-e15732c4e467\") "
Sep 16 19:08:32.683244 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.683223    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a3fb07-714c-43e9-b93b-e15732c4e467-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31a3fb07-714c-43e9-b93b-e15732c4e467" (UID: "31a3fb07-714c-43e9-b93b-e15732c4e467"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:32.683311 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.683293    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31a3fb07-714c-43e9-b93b-e15732c4e467" (UID: "31a3fb07-714c-43e9-b93b-e15732c4e467"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:32.683350 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.683330    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31a3fb07-714c-43e9-b93b-e15732c4e467" (UID: "31a3fb07-714c-43e9-b93b-e15732c4e467"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:32.685090 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.685064    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31a3fb07-714c-43e9-b93b-e15732c4e467-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31a3fb07-714c-43e9-b93b-e15732c4e467" (UID: "31a3fb07-714c-43e9-b93b-e15732c4e467"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:32.685173 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.685130    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-api-access-k5gz6" (OuterVolumeSpecName: "kube-api-access-k5gz6") pod "31a3fb07-714c-43e9-b93b-e15732c4e467" (UID: "31a3fb07-714c-43e9-b93b-e15732c4e467"). InnerVolumeSpecName "kube-api-access-k5gz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:32.784076 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.784045    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31a3fb07-714c-43e9-b93b-e15732c4e467-kserve-provision-location\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:08:32.784076 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.784072    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:08:32.784076 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.784083    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31a3fb07-714c-43e9-b93b-e15732c4e467-cabundle-cert\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:08:32.784288 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.784095    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31a3fb07-714c-43e9-b93b-e15732c4e467-proxy-tls\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:08:32.784288 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.784102    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5gz6\" (UniqueName: \"kubernetes.io/projected/31a3fb07-714c-43e9-b93b-e15732c4e467-kube-api-access-k5gz6\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:08:32.966532 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.966491    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"]
Sep 16 19:08:32.970549 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:32.970502    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-27mrk"]
Sep 16 19:08:34.447111 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:34.447080    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" path="/var/lib/kubelet/pods/31a3fb07-714c-43e9-b93b-e15732c4e467/volumes"
Sep 16 19:08:39.352153 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352113    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"]
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352440    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352450    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352461    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="storage-initializer"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352467    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="storage-initializer"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352474    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kube-rbac-proxy"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352482    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kube-rbac-proxy"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352491    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kube-rbac-proxy"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352496    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kube-rbac-proxy"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352503    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352508    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352534    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="storage-initializer"
Sep 16 19:08:39.352579 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.352540    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="storage-initializer"
Sep 16 19:08:39.353174 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.353144    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kserve-container"
Sep 16 19:08:39.353357 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.353344    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f9d2db-6097-4ea6-9de7-70304a5ee0a5" containerName="kube-rbac-proxy"
Sep 16 19:08:39.353439 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.353429    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kube-rbac-proxy"
Sep 16 19:08:39.353558 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.353546    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a3fb07-714c-43e9-b93b-e15732c4e467" containerName="kserve-container"
Sep 16 19:08:39.360610 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.360583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.362915 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.362892    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:08:39.363045 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.363026    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:08:39.363216 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.363198    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:08:39.363281 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.363268    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:08:39.363613 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.363601    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:08:39.363685 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.363601    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:08:39.363685 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.363665    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 19:08:39.371805 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.371785    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"]
Sep 16 19:08:39.436759 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.436725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksw9\" (UniqueName: \"kubernetes.io/projected/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-api-access-mksw9\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.436927 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.436782    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.436927 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.436839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.436927 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.436882    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.436927 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.436902    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.537392 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.537350    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.537623 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.537410    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.537623 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.537454    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.537623 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.537485    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.537623 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.537540    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksw9\" (UniqueName: \"kubernetes.io/projected/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-api-access-mksw9\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.537623 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:08:39.537615    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:08:39.537896 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:08:39.537696    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-proxy-tls podName:807b9513-8c4a-4fe6-b0ba-11b32bcb1648 nodeName:}" failed. No retries permitted until 2026-09-16 19:08:40.037673484 +0000 UTC m=+1394.108770687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-proxy-tls") pod "isvc-canary-promote-predictor-649dd98487-khl5g" (UID: "807b9513-8c4a-4fe6-b0ba-11b32bcb1648") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:08:39.538055 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.538034    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.538267 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.538244    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.538324 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.538300    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:39.546451 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:39.546423    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksw9\" (UniqueName: \"kubernetes.io/projected/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-api-access-mksw9\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:40.041629 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:40.041586    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:40.044300 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:40.044265    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-khl5g\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:40.271748 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:40.271710    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:40.552877 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:40.552850    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"]
Sep 16 19:08:40.555335 ip-10-0-128-222 kubenswrapper[2572]: W0916 19:08:40.555298    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807b9513_8c4a_4fe6_b0ba_11b32bcb1648.slice/crio-dd7c18d9fd9a5affee73d463973a0ee9d3371a037d5be16a1e295a977c1f4563 WatchSource:0}: Error finding container dd7c18d9fd9a5affee73d463973a0ee9d3371a037d5be16a1e295a977c1f4563: Status 404 returned error can't find the container with id dd7c18d9fd9a5affee73d463973a0ee9d3371a037d5be16a1e295a977c1f4563
Sep 16 19:08:40.680309 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:40.680264    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" event={"ID":"807b9513-8c4a-4fe6-b0ba-11b32bcb1648","Type":"ContainerStarted","Data":"bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e"}
Sep 16 19:08:40.680500 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:40.680328    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" event={"ID":"807b9513-8c4a-4fe6-b0ba-11b32bcb1648","Type":"ContainerStarted","Data":"dd7c18d9fd9a5affee73d463973a0ee9d3371a037d5be16a1e295a977c1f4563"}
Sep 16 19:08:44.699404 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:44.699369    2572 generic.go:356] "Generic (PLEG): container finished" podID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerID="bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e" exitCode=0
Sep 16 19:08:44.699831 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:44.699424    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" event={"ID":"807b9513-8c4a-4fe6-b0ba-11b32bcb1648","Type":"ContainerDied","Data":"bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e"}
Sep 16 19:08:45.706223 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:45.706183    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" event={"ID":"807b9513-8c4a-4fe6-b0ba-11b32bcb1648","Type":"ContainerStarted","Data":"aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a"}
Sep 16 19:08:45.706675 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:45.706257    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" event={"ID":"807b9513-8c4a-4fe6-b0ba-11b32bcb1648","Type":"ContainerStarted","Data":"eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c"}
Sep 16 19:08:45.706675 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:45.706543    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:45.706675 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:45.706599    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:45.708232 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:45.708203    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:08:45.736121 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:45.736060    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podStartSLOduration=3.143319935 podStartE2EDuration="6.736044015s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.59272408s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:45.734906866 +0000 UTC m=+1399.806004094" watchObservedRunningTime="2026-09-16 19:08:45.736044015 +0000 UTC m=+1399.807141239"
Sep 16 19:08:46.711178 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:46.711140    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:08:51.717574 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:51.717545    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:08:51.718059 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:08:51.718032    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:09:01.718234 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:09:01.718196    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:09:11.718012 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:09:11.717960    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:09:21.718712 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:09:21.718628    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:09:31.719041 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:09:31.718999    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:09:41.718296 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:09:41.718254    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:09:51.718279 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:09:51.718236    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:10:01.718686 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:01.718646    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:10:09.425790 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:09.425749    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"]
Sep 16 19:10:09.426407 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:09.426045    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" containerID="cri-o://eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c" gracePeriod=30
Sep 16 19:10:09.426407 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:09.426079    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kube-rbac-proxy" containerID="cri-o://aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a" gracePeriod=30
Sep 16 19:10:10.089337 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:10.089303    2572 generic.go:356] "Generic (PLEG): container finished" podID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerID="aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a" exitCode=2
Sep 16 19:10:10.089532 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:10.089363    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" event={"ID":"807b9513-8c4a-4fe6-b0ba-11b32bcb1648","Type":"ContainerDied","Data":"aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a"}
Sep 16 19:10:11.711593 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:11.711539    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 16 19:10:11.718752 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:11.718721    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 16 19:10:14.375924 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.375900    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:10:14.388392 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.388364    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kserve-provision-location\") pod \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") "
Sep 16 19:10:14.388577 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.388426    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-rbac-proxy-sar-config\") pod \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") "
Sep 16 19:10:14.388577 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.388449    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mksw9\" (UniqueName: \"kubernetes.io/projected/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-api-access-mksw9\") pod \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") "
Sep 16 19:10:14.388577 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.388480    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-proxy-tls\") pod \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") "
Sep 16 19:10:14.388577 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.388535    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-cabundle-cert\") pod \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\" (UID: \"807b9513-8c4a-4fe6-b0ba-11b32bcb1648\") "
Sep 16 19:10:14.388795 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.388739    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "807b9513-8c4a-4fe6-b0ba-11b32bcb1648" (UID: "807b9513-8c4a-4fe6-b0ba-11b32bcb1648"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:14.388908 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.388795    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "807b9513-8c4a-4fe6-b0ba-11b32bcb1648" (UID: "807b9513-8c4a-4fe6-b0ba-11b32bcb1648"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:14.388988 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.388968    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "807b9513-8c4a-4fe6-b0ba-11b32bcb1648" (UID: "807b9513-8c4a-4fe6-b0ba-11b32bcb1648"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:14.390690 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.390667    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-api-access-mksw9" (OuterVolumeSpecName: "kube-api-access-mksw9") pod "807b9513-8c4a-4fe6-b0ba-11b32bcb1648" (UID: "807b9513-8c4a-4fe6-b0ba-11b32bcb1648"). InnerVolumeSpecName "kube-api-access-mksw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:14.390833 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.390807    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "807b9513-8c4a-4fe6-b0ba-11b32bcb1648" (UID: "807b9513-8c4a-4fe6-b0ba-11b32bcb1648"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:14.489500 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.489458    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-proxy-tls\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:10:14.489500 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.489491    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-cabundle-cert\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:10:14.489500 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.489502    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kserve-provision-location\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:10:14.489751 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.489534    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:10:14.489751 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:14.489543    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mksw9\" (UniqueName: \"kubernetes.io/projected/807b9513-8c4a-4fe6-b0ba-11b32bcb1648-kube-api-access-mksw9\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:10:15.113341 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.113306    2572 generic.go:356] "Generic (PLEG): container finished" podID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerID="eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c" exitCode=0
Sep 16 19:10:15.113591 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.113368    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"
Sep 16 19:10:15.113591 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.113371    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" event={"ID":"807b9513-8c4a-4fe6-b0ba-11b32bcb1648","Type":"ContainerDied","Data":"eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c"}
Sep 16 19:10:15.113591 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.113496    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g" event={"ID":"807b9513-8c4a-4fe6-b0ba-11b32bcb1648","Type":"ContainerDied","Data":"dd7c18d9fd9a5affee73d463973a0ee9d3371a037d5be16a1e295a977c1f4563"}
Sep 16 19:10:15.113591 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.113554    2572 scope.go:117] "RemoveContainer" containerID="aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a"
Sep 16 19:10:15.123470 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.123442    2572 scope.go:117] "RemoveContainer" containerID="eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c"
Sep 16 19:10:15.131075 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.131055    2572 scope.go:117] "RemoveContainer" containerID="bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e"
Sep 16 19:10:15.134735 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.134714    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"]
Sep 16 19:10:15.139113 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.139096    2572 scope.go:117] "RemoveContainer" containerID="aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a"
Sep 16 19:10:15.139174 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.139107    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-khl5g"]
Sep 16 19:10:15.139392 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:10:15.139371    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a\": container with ID starting with aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a not found: ID does not exist" containerID="aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a"
Sep 16 19:10:15.139440 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.139400    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a"} err="failed to get container status \"aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a\": rpc error: code = NotFound desc = could not find container \"aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a\": container with ID starting with aa43427ce26af51a85d8ca683822762dfccd46260324b2197bc4992f68e9f43a not found: ID does not exist"
Sep 16 19:10:15.139440 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.139415    2572 scope.go:117] "RemoveContainer" containerID="eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c"
Sep 16 19:10:15.139677 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:10:15.139660    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c\": container with ID starting with eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c not found: ID does not exist" containerID="eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c"
Sep 16 19:10:15.139730 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.139680    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c"} err="failed to get container status \"eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c\": rpc error: code = NotFound desc = could not find container \"eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c\": container with ID starting with eb0f2eee7d085f3c85ea93a6532fed56afed82c7d08a040352d0ad7abd3def3c not found: ID does not exist"
Sep 16 19:10:15.139730 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.139693    2572 scope.go:117] "RemoveContainer" containerID="bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e"
Sep 16 19:10:15.139924 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:10:15.139910    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e\": container with ID starting with bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e not found: ID does not exist" containerID="bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e"
Sep 16 19:10:15.139970 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:15.139927    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e"} err="failed to get container status \"bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e\": rpc error: code = NotFound desc = could not find container \"bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e\": container with ID starting with bf651996327dba0b946ca2dec85adcf97cd5ea28d2483dbba80d6696564b893e not found: ID does not exist"
Sep 16 19:10:16.447876 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:10:16.447844    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" path="/var/lib/kubelet/pods/807b9513-8c4a-4fe6-b0ba-11b32bcb1648/volumes"
Sep 16 19:11:54.277428 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.277387    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"]
Sep 16 19:11:54.280009 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.277910    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container"
Sep 16 19:11:54.280009 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.277929    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container"
Sep 16 19:11:54.280009 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.277946    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kube-rbac-proxy"
Sep 16 19:11:54.280009 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.277952    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kube-rbac-proxy"
Sep 16 19:11:54.280009 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.277963    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="storage-initializer"
Sep 16 19:11:54.280009 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.277969    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="storage-initializer"
Sep 16 19:11:54.280009 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.278026    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kserve-container"
Sep 16 19:11:54.280009 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.278036    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="807b9513-8c4a-4fe6-b0ba-11b32bcb1648" containerName="kube-rbac-proxy"
Sep 16 19:11:54.280938 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.280918    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.284576 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.284551    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 16 19:11:54.285407 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.285375    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:11:54.285528 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.285440    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:11:54.285528 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.285481    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 16 19:11:54.285822 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.285809    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\""
Sep 16 19:11:54.296039 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.296013    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"]
Sep 16 19:11:54.413465 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.413424    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71bd085e-94af-46bc-9e18-c8aff7a0d879-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-wlklp\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.413673 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.413476    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-wlklp\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.413673 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.413502    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkpb\" (UniqueName: \"kubernetes.io/projected/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-api-access-8jkpb\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-wlklp\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.514845 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.514804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-wlklp\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.514845 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.514848    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkpb\" (UniqueName: \"kubernetes.io/projected/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-api-access-8jkpb\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-wlklp\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.515070 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.514923    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71bd085e-94af-46bc-9e18-c8aff7a0d879-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-wlklp\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.515487 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.515467    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-wlklp\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.517713 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.517691    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71bd085e-94af-46bc-9e18-c8aff7a0d879-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-wlklp\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.524659 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.524632    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkpb\" (UniqueName: \"kubernetes.io/projected/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-api-access-8jkpb\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-wlklp\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.591728 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.591629    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:54.874868 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.874783    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"]
Sep 16 19:11:54.879062 ip-10-0-128-222 kubenswrapper[2572]: W0916 19:11:54.879035    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bd085e_94af_46bc_9e18_c8aff7a0d879.slice/crio-217a6a114b112070fcec9dd6713ff912cbfa4442dce4eac830ffde9ff2eea628 WatchSource:0}: Error finding container 217a6a114b112070fcec9dd6713ff912cbfa4442dce4eac830ffde9ff2eea628: Status 404 returned error can't find the container with id 217a6a114b112070fcec9dd6713ff912cbfa4442dce4eac830ffde9ff2eea628
Sep 16 19:11:54.881031 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:54.881011    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:11:55.555207 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:55.555176    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp" event={"ID":"71bd085e-94af-46bc-9e18-c8aff7a0d879","Type":"ContainerStarted","Data":"217a6a114b112070fcec9dd6713ff912cbfa4442dce4eac830ffde9ff2eea628"}
Sep 16 19:11:56.561118 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:56.561087    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp" event={"ID":"71bd085e-94af-46bc-9e18-c8aff7a0d879","Type":"ContainerStarted","Data":"6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b"}
Sep 16 19:11:56.561576 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:56.561141    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp" event={"ID":"71bd085e-94af-46bc-9e18-c8aff7a0d879","Type":"ContainerStarted","Data":"9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016"}
Sep 16 19:11:56.561576 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:56.561221    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:56.581266 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:56.581211    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp" podStartSLOduration=1.478197347 podStartE2EDuration="2.581194997s" totalImagesPullingTime="1.10299765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:56.580124326 +0000 UTC m=+1590.651221550" watchObservedRunningTime="2026-09-16 19:11:56.581194997 +0000 UTC m=+1590.652292221"
Sep 16 19:11:57.565394 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:57.565367    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:11:57.567233 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:11:57.567214    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:12:04.584685 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:12:04.584654    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:13:49.333203 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.333119    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-wlklp_71bd085e-94af-46bc-9e18-c8aff7a0d879/kserve-container/0.log"
Sep 16 19:13:49.596674 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.596590    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"]
Sep 16 19:13:49.596889 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.596841    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp" podUID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerName="kserve-container" containerID="cri-o://9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016" gracePeriod=30
Sep 16 19:13:49.596997 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.596879    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp" podUID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerName="kube-rbac-proxy" containerID="cri-o://6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b" gracePeriod=30
Sep 16 19:13:49.840673 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.840651    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:13:49.950735 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.950704    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71bd085e-94af-46bc-9e18-c8aff7a0d879-proxy-tls\") pod \"71bd085e-94af-46bc-9e18-c8aff7a0d879\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") "
Sep 16 19:13:49.950893 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.950750    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-rbac-proxy-sar-config\") pod \"71bd085e-94af-46bc-9e18-c8aff7a0d879\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") "
Sep 16 19:13:49.950893 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.950815    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jkpb\" (UniqueName: \"kubernetes.io/projected/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-api-access-8jkpb\") pod \"71bd085e-94af-46bc-9e18-c8aff7a0d879\" (UID: \"71bd085e-94af-46bc-9e18-c8aff7a0d879\") "
Sep 16 19:13:49.951140 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.951117    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71bd085e-94af-46bc-9e18-c8aff7a0d879" (UID: "71bd085e-94af-46bc-9e18-c8aff7a0d879"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:49.952974 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.952946    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-api-access-8jkpb" (OuterVolumeSpecName: "kube-api-access-8jkpb") pod "71bd085e-94af-46bc-9e18-c8aff7a0d879" (UID: "71bd085e-94af-46bc-9e18-c8aff7a0d879"). InnerVolumeSpecName "kube-api-access-8jkpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:49.953094 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:49.952982    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71bd085e-94af-46bc-9e18-c8aff7a0d879-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71bd085e-94af-46bc-9e18-c8aff7a0d879" (UID: "71bd085e-94af-46bc-9e18-c8aff7a0d879"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:50.051438 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.051406    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkpb\" (UniqueName: \"kubernetes.io/projected/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-api-access-8jkpb\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:13:50.051438 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.051436    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71bd085e-94af-46bc-9e18-c8aff7a0d879-proxy-tls\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:13:50.051438 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.051445    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71bd085e-94af-46bc-9e18-c8aff7a0d879-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-222.ec2.internal\" DevicePath \"\""
Sep 16 19:13:50.063682 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.063643    2572 generic.go:356] "Generic (PLEG): container finished" podID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerID="6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b" exitCode=2
Sep 16 19:13:50.063682 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.063677    2572 generic.go:356] "Generic (PLEG): container finished" podID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerID="9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016" exitCode=2
Sep 16 19:13:50.063864 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.063704    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"
Sep 16 19:13:50.063864 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.063731    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp" event={"ID":"71bd085e-94af-46bc-9e18-c8aff7a0d879","Type":"ContainerDied","Data":"6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b"}
Sep 16 19:13:50.063864 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.063792    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp" event={"ID":"71bd085e-94af-46bc-9e18-c8aff7a0d879","Type":"ContainerDied","Data":"9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016"}
Sep 16 19:13:50.063864 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.063814    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp" event={"ID":"71bd085e-94af-46bc-9e18-c8aff7a0d879","Type":"ContainerDied","Data":"217a6a114b112070fcec9dd6713ff912cbfa4442dce4eac830ffde9ff2eea628"}
Sep 16 19:13:50.063864 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.063837    2572 scope.go:117] "RemoveContainer" containerID="6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b"
Sep 16 19:13:50.072417 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.072398    2572 scope.go:117] "RemoveContainer" containerID="9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016"
Sep 16 19:13:50.079989 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.079967    2572 scope.go:117] "RemoveContainer" containerID="6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b"
Sep 16 19:13:50.080257 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:13:50.080238    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b\": container with ID starting with 6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b not found: ID does not exist" containerID="6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b"
Sep 16 19:13:50.080340 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.080269    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b"} err="failed to get container status \"6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b\": rpc error: code = NotFound desc = could not find container \"6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b\": container with ID starting with 6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b not found: ID does not exist"
Sep 16 19:13:50.080340 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.080296    2572 scope.go:117] "RemoveContainer" containerID="9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016"
Sep 16 19:13:50.080581 ip-10-0-128-222 kubenswrapper[2572]: E0916 19:13:50.080563    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016\": container with ID starting with 9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016 not found: ID does not exist" containerID="9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016"
Sep 16 19:13:50.080625 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.080587    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016"} err="failed to get container status \"9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016\": rpc error: code = NotFound desc = could not find container \"9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016\": container with ID starting with 9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016 not found: ID does not exist"
Sep 16 19:13:50.080625 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.080602    2572 scope.go:117] "RemoveContainer" containerID="6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b"
Sep 16 19:13:50.080841 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.080823    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b"} err="failed to get container status \"6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b\": rpc error: code = NotFound desc = could not find container \"6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b\": container with ID starting with 6297a6b989e4ee2bfb69a6075006d007920ceaea824a6f48bb790e28717ee82b not found: ID does not exist"
Sep 16 19:13:50.080889 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.080841    2572 scope.go:117] "RemoveContainer" containerID="9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016"
Sep 16 19:13:50.081053 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.081034    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016"} err="failed to get container status \"9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016\": rpc error: code = NotFound desc = could not find container \"9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016\": container with ID starting with 9f689c6f37047b6288003767dc3348428628d84eca17cda2dd1858bdd3f8f016 not found: ID does not exist"
Sep 16 19:13:50.084915 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.084894    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"]
Sep 16 19:13:50.089848 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.089827    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-wlklp"]
Sep 16 19:13:50.448106 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:13:50.448075    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71bd085e-94af-46bc-9e18-c8aff7a0d879" path="/var/lib/kubelet/pods/71bd085e-94af-46bc-9e18-c8aff7a0d879/volumes"
Sep 16 19:14:25.341192 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:25.341158    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sll74_ce358ba2-d29f-4719-b826-c8726759568c/global-pull-secret-syncer/0.log"
Sep 16 19:14:25.441808 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:25.441782    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kbf8w_fd4d4a63-f6db-45d5-8a3f-c80fc416446e/konnectivity-agent/0.log"
Sep 16 19:14:25.491294 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:25.491269    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-222.ec2.internal_254bd750f9fd800e2c1818cb4a5d50a1/haproxy/0.log"
Sep 16 19:14:29.042161 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:29.042120    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-fnzwg_ab856818-b5aa-4b90-b7ac-fc9743acf780/cluster-monitoring-operator/0.log"
Sep 16 19:14:29.229703 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:29.229677    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-jmmcz_4119327c-86c6-4e55-b4e5-9c47b4219fd2/monitoring-plugin/0.log"
Sep 16 19:14:29.394839 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:29.394761    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7xcg9_42466999-4bb2-480c-941b-6365812b70f2/node-exporter/0.log"
Sep 16 19:14:29.421253 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:29.421226    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7xcg9_42466999-4bb2-480c-941b-6365812b70f2/kube-rbac-proxy/0.log"
Sep 16 19:14:29.460413 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:29.460387    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7xcg9_42466999-4bb2-480c-941b-6365812b70f2/init-textfile/0.log"
Sep 16 19:14:29.988347 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:29.988312    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wvnxt_f68efdfd-99b2-4954-9ed3-d295110703b7/prometheus-operator/0.log"
Sep 16 19:14:30.017519 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:30.017487    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wvnxt_f68efdfd-99b2-4954-9ed3-d295110703b7/kube-rbac-proxy/0.log"
Sep 16 19:14:30.170931 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:30.170903    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76664c6f65-rw2nd_64b66e25-8961-4710-b06a-04382d3fb8e9/thanos-query/0.log"
Sep 16 19:14:30.197259 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:30.197237    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76664c6f65-rw2nd_64b66e25-8961-4710-b06a-04382d3fb8e9/kube-rbac-proxy-web/0.log"
Sep 16 19:14:30.219795 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:30.219772    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76664c6f65-rw2nd_64b66e25-8961-4710-b06a-04382d3fb8e9/kube-rbac-proxy/0.log"
Sep 16 19:14:30.257239 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:30.257158    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76664c6f65-rw2nd_64b66e25-8961-4710-b06a-04382d3fb8e9/prom-label-proxy/0.log"
Sep 16 19:14:30.286799 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:30.286772    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76664c6f65-rw2nd_64b66e25-8961-4710-b06a-04382d3fb8e9/kube-rbac-proxy-rules/0.log"
Sep 16 19:14:30.323722 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:30.323696    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76664c6f65-rw2nd_64b66e25-8961-4710-b06a-04382d3fb8e9/kube-rbac-proxy-metrics/0.log"
Sep 16 19:14:32.885057 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:32.885024    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-95c57_d1108c81-4a31-4a0f-8856-419c30c68edb/download-server/0.log"
Sep 16 19:14:33.856147 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.856115    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"]
Sep 16 19:14:33.856461 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.856449    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerName="kserve-container"
Sep 16 19:14:33.856505 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.856463    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerName="kserve-container"
Sep 16 19:14:33.856505 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.856472    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerName="kube-rbac-proxy"
Sep 16 19:14:33.856505 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.856477    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerName="kube-rbac-proxy"
Sep 16 19:14:33.856627 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.856544    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerName="kserve-container"
Sep 16 19:14:33.856627 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.856552    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="71bd085e-94af-46bc-9e18-c8aff7a0d879" containerName="kube-rbac-proxy"
Sep 16 19:14:33.859544 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.859495    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:33.861695 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.861673    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68wb5\"/\"kube-root-ca.crt\""
Sep 16 19:14:33.861829 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.861716    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68wb5\"/\"openshift-service-ca.crt\""
Sep 16 19:14:33.861829 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.861746    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-68wb5\"/\"default-dockercfg-fd5h5\""
Sep 16 19:14:33.875919 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.875894    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"]
Sep 16 19:14:33.915575 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.915548    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-lib-modules\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:33.915575 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.915579    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-sys\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:33.916008 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.915609    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-podres\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:33.916008 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.915641    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xjm\" (UniqueName: \"kubernetes.io/projected/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-kube-api-access-d6xjm\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:33.916008 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:33.915669    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-proc\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.016427 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.016373    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-lib-modules\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.016427 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.016428    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-sys\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.016722 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.016473    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-podres\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.016722 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.016494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xjm\" (UniqueName: \"kubernetes.io/projected/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-kube-api-access-d6xjm\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.016722 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.016590    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-lib-modules\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.016722 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.016610    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-proc\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.016722 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.016591    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-sys\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.016722 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.016632    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-podres\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.016722 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.016655    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-proc\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.024767 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.024744    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xjm\" (UniqueName: \"kubernetes.io/projected/08d0c4ea-7a0a-45d3-bcf0-49303885bea6-kube-api-access-d6xjm\") pod \"perf-node-gather-daemonset-rqnzw\" (UID: \"08d0c4ea-7a0a-45d3-bcf0-49303885bea6\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.169138 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.169028    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:34.452586 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.452564    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"]
Sep 16 19:14:34.455267 ip-10-0-128-222 kubenswrapper[2572]: W0916 19:14:34.455238    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod08d0c4ea_7a0a_45d3_bcf0_49303885bea6.slice/crio-7066bff4772e5f7da5aa9dc6409192dc82475d680c95a66049feb76726b28ab3 WatchSource:0}: Error finding container 7066bff4772e5f7da5aa9dc6409192dc82475d680c95a66049feb76726b28ab3: Status 404 returned error can't find the container with id 7066bff4772e5f7da5aa9dc6409192dc82475d680c95a66049feb76726b28ab3
Sep 16 19:14:34.483505 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.483486    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp57f_b713bc4f-84ae-422e-8c29-0e37ac9f5f8d/dns/0.log"
Sep 16 19:14:34.523288 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.523254    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp57f_b713bc4f-84ae-422e-8c29-0e37ac9f5f8d/kube-rbac-proxy/0.log"
Sep 16 19:14:34.571492 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:34.571457    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fjbx_b0dcc2f3-5e91-4dd1-a54d-9a1fe99eb368/dns-node-resolver/0.log"
Sep 16 19:14:35.180056 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:35.180028    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85967c5989-mn8js_5d793038-dc8f-45d8-a3b9-af84ae589c23/registry/0.log"
Sep 16 19:14:35.187180 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:35.187156    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85967c5989-mn8js_5d793038-dc8f-45d8-a3b9-af84ae589c23/registry/1.log"
Sep 16 19:14:35.232944 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:35.232914    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xtfbc_67557c8c-a566-40ca-b64c-8760f4fbbbe7/node-ca/0.log"
Sep 16 19:14:35.262006 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:35.261973    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw" event={"ID":"08d0c4ea-7a0a-45d3-bcf0-49303885bea6","Type":"ContainerStarted","Data":"ed9802b8f35ae385440085dd16be55aaad7c53aacbb8e9bc4f071ec85994346b"}
Sep 16 19:14:35.262160 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:35.262030    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw" event={"ID":"08d0c4ea-7a0a-45d3-bcf0-49303885bea6","Type":"ContainerStarted","Data":"7066bff4772e5f7da5aa9dc6409192dc82475d680c95a66049feb76726b28ab3"}
Sep 16 19:14:35.262160 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:35.262057    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:35.291533 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:35.291462    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw" podStartSLOduration=2.291446258 podStartE2EDuration="2.291446258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:35.290556958 +0000 UTC m=+1749.361654182" watchObservedRunningTime="2026-09-16 19:14:35.291446258 +0000 UTC m=+1749.362543481"
Sep 16 19:14:36.003911 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:36.003881    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f9dc5957d-6j7sr_783e7bea-f22c-4f77-9958-facda1f07bd9/router/0.log"
Sep 16 19:14:36.396282 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:36.396211    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v6bsb_49314118-b6b2-47ce-b1c4-e5dd1140a71a/serve-healthcheck-canary/0.log"
Sep 16 19:14:36.846292 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:36.846262    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l97v6_7df4f878-1807-43fd-acdd-3a6cff6bc930/kube-rbac-proxy/0.log"
Sep 16 19:14:36.868269 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:36.868244    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l97v6_7df4f878-1807-43fd-acdd-3a6cff6bc930/exporter/0.log"
Sep 16 19:14:36.891117 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:36.891094    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l97v6_7df4f878-1807-43fd-acdd-3a6cff6bc930/extractor/0.log"
Sep 16 19:14:39.266017 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:39.265980    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-lsrhv_170ed54a-d944-49a3-9023-1578e2a6fb28/manager/0.log"
Sep 16 19:14:39.399001 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:39.398967    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-mb9qw_719b95c2-5fff-4eed-87fd-716adc1339a8/manager/0.log"
Sep 16 19:14:41.277639 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:41.277607    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-rqnzw"
Sep 16 19:14:43.768860 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:43.768835    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-s7h8k_b085dfcf-c262-451c-8e7e-1bf92f7164f6/migrator/0.log"
Sep 16 19:14:43.797679 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:43.797654    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-s7h8k_b085dfcf-c262-451c-8e7e-1bf92f7164f6/graceful-termination/0.log"
Sep 16 19:14:44.298404 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:44.298376    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-n825t_71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff/kube-storage-version-migrator-operator/1.log"
Sep 16 19:14:44.299409 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:44.299362    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-n825t_71b869ae-cd5b-4bc0-89b7-6164c7dbb0ff/kube-storage-version-migrator-operator/0.log"
Sep 16 19:14:45.594927 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:45.594900    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdj6m_fb01d6a2-e6d0-41b6-944a-7158baacf6a0/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:14:45.632478 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:45.632449    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdj6m_fb01d6a2-e6d0-41b6-944a-7158baacf6a0/egress-router-binary-copy/0.log"
Sep 16 19:14:45.676553 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:45.676483    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdj6m_fb01d6a2-e6d0-41b6-944a-7158baacf6a0/cni-plugins/0.log"
Sep 16 19:14:45.787061 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:45.787034    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdj6m_fb01d6a2-e6d0-41b6-944a-7158baacf6a0/bond-cni-plugin/0.log"
Sep 16 19:14:45.887620 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:45.887572    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdj6m_fb01d6a2-e6d0-41b6-944a-7158baacf6a0/routeoverride-cni/0.log"
Sep 16 19:14:45.938606 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:45.938537    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdj6m_fb01d6a2-e6d0-41b6-944a-7158baacf6a0/whereabouts-cni-bincopy/0.log"
Sep 16 19:14:45.989441 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:45.989387    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fdj6m_fb01d6a2-e6d0-41b6-944a-7158baacf6a0/whereabouts-cni/0.log"
Sep 16 19:14:46.613925 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:46.613893    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ztpjp_53a203ba-b578-4d23-be8b-de7a9208cef2/kube-multus/0.log"
Sep 16 19:14:46.637693 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:46.637669    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-856sw_2c4cd60c-ec4c-4284-b45e-bb39c7786cf8/network-metrics-daemon/0.log"
Sep 16 19:14:46.657037 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:46.657012    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-856sw_2c4cd60c-ec4c-4284-b45e-bb39c7786cf8/kube-rbac-proxy/0.log"
Sep 16 19:14:48.168893 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:48.168864    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wc29d_ceb6ae6a-1648-41fa-8013-7637311b5dea/ovn-controller/0.log"
Sep 16 19:14:48.188877 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:48.188818    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wc29d_ceb6ae6a-1648-41fa-8013-7637311b5dea/ovn-acl-logging/0.log"
Sep 16 19:14:48.212665 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:48.212638    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wc29d_ceb6ae6a-1648-41fa-8013-7637311b5dea/kube-rbac-proxy-node/0.log"
Sep 16 19:14:48.232319 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:48.232290    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wc29d_ceb6ae6a-1648-41fa-8013-7637311b5dea/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:14:48.250083 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:48.250058    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wc29d_ceb6ae6a-1648-41fa-8013-7637311b5dea/northd/0.log"
Sep 16 19:14:48.269903 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:48.269873    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wc29d_ceb6ae6a-1648-41fa-8013-7637311b5dea/nbdb/0.log"
Sep 16 19:14:48.291472 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:48.291447    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wc29d_ceb6ae6a-1648-41fa-8013-7637311b5dea/sbdb/0.log"
Sep 16 19:14:48.329836 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:48.329810    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wc29d_ceb6ae6a-1648-41fa-8013-7637311b5dea/ovnkube-controller/0.log"
Sep 16 19:14:49.394297 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:49.394267    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-sz5vt_0c903d9e-cd3d-4818-be5d-299cfd855b2e/network-check-target-container/0.log"
Sep 16 19:14:50.333357 ip-10-0-128-222 kubenswrapper[2572]: I0916 19:14:50.333327    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jkvjw_31f543ac-8883-46d0-a052-128752bbe6f0/iptables-alerter/0.log"