Sep 16 18:40:28.555597 ip-10-0-128-29 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:28.855036 ip-10-0-128-29 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:28.855036 ip-10-0-128-29 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:28.855036 ip-10-0-128-29 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:28.855036 ip-10-0-128-29 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:28.855036 ip-10-0-128-29 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:28.856522 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.856459    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:28.859167 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859152    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:28.859167 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859167    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859170    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859175    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859178    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859181    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859183    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859186    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859189    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859192    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859195    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859197    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859200    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859203    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859205    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859208    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859210    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859213    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859216    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859218    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859226    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:28.859229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859228    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859231    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859234    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859237    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859240    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859243    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859246    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859248    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859251    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859253    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859256    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859259    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859261    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859263    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859266    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859269    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859271    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859274    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859276    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859279    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:28.859816 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859282    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859287    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859303    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859307    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859310    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859313    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859316    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859318    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859322    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859324    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859327    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859329    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859332    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859334    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859338    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859341    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859343    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859346    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859349    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:28.860569 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859351    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859354    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859356    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859359    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859362    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859365    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859367    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859371    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859375    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859378    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859381    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859384    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859386    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859389    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859392    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859394    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859397    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859399    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859402    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:28.861200 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859404    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859408    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859410    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859413    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859416    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859419    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859421    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859424    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859427    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859430    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859433    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859435    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859438    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859442    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859445    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859449    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859451    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859454    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859457    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.859460    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:28.861676 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861397    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861408    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861412    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861415    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861418    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861421    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861424    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861428    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861431    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861434    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861438    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861442    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861445    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861447    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861451    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861453    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861456    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861459    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861462    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861464    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:28.862160 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861468    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861471    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861474    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861477    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861480    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861482    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861485    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861488    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861491    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861495    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861498    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861501    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861504    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861506    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861509    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861511    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861514    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861516    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861519    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861522    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:28.862700 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861524    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861527    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861529    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861533    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861536    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861539    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861543    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861548    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861552    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861555    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861557    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861560    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861562    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861565    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861567    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861570    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861572    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861575    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861578    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:28.863193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861580    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861583    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861585    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861588    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861591    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861593    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861596    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861598    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861601    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861604    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861606    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861609    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861611    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861614    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861617    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861619    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861622    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861625    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861627    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861630    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:28.863697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861632    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861635    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861637    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861641    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861644    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861647    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861649    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861652    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861654    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861657    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861659    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861662    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861664    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861667    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861669    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861672    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861674    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861677    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861679    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.861682    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:28.864193 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862044    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862048    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862051    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862054    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862057    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862060    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862063    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862065    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862068    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862071    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862075    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862079    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862081    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862084    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862086    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862089    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862092    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862095    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862097    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862100    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:28.864697 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862102    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862105    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862107    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862110    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862112    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862115    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862117    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862120    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862122    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862125    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862127    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862130    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862132    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862135    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862144    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862151    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862153    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862156    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862159    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862162    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:28.865196 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862165    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862168    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862171    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862173    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862176    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862179    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862181    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862184    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862187    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862190    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862193    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862195    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862200    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862204    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862207    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862210    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862213    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862215    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862218    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862221    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:28.865703 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862224    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862227    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862229    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862232    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862235    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862238    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862240    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862243    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862248    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862250    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862253    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862256    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862260    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862264    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862266    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862269    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862271    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862274    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862277    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:28.866181 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862279    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862282    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862284    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862287    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862303    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862306    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862308    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862311    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862313    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862316    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862318    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862321    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862323    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862326    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862328    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862331    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862333    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862336    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862339    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.862341    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:28.866707 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862415    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862451    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862457    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862461    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862465    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862469    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862473    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862477    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862480    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862483    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862486    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862490    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862493    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862496    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862499    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862502    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862506    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862509    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862514    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862517    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862520    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862523    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862526    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862529    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:28.867194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862532    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862535    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862539    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862542    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862545    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862547    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862551    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862554    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862558    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862563    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862565    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862568    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862571    2561 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862583    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862588    2561 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862591    2561 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862594    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862597    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862600    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862604    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862607    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862609    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862612    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862615    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862618    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:28.867801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862623    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862626    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862629    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862631    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862634    2561 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862638    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862641    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862644    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862647    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862650    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862653    2561 flags.go:64] FLAG: --help="false"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862656    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-128-29.ec2.internal"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862659    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862662    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862664    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862668    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862673    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862676    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862679    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862682    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862686    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862689    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862692    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862695    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:28.868514 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862697    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862700    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862704    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862707    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862710    2561 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862713    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862716    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862719    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862724    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862728    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862731    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862733    2561 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862736    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862739    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862742    2561 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862745    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862749    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862752    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862756    2561 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862758    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862761    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862764    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862767    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862770    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862774    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:28.869181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862777    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862784    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862787    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862791    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862794    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862797    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862802    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862805    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862808    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862810    2561 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862814    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862817    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03020b910531ff95e"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862820    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862823    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862826    2561 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862828    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862832    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862835    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862838    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862841    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862845    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862848    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862851    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862853    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862856    2561 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:28.869809 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862859    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862862    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862865    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862868    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862871    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862874    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862877    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862881    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862884    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862886    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862890    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862893    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862896    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862899    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862902    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862907    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862910    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862913    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862917    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862920    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862922    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862925    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862928    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862931    2561 flags.go:64] FLAG: --v="2"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862934    2561 flags.go:64] FLAG: --version="false"
Sep 16 18:40:28.870441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862939    2561 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862942    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.862945    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863076    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863080    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863083    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863085    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863088    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863090    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863093    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863096    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863098    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863101    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863103    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863107    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863110    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863112    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863116    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863119    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:28.871067 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863121    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863125    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863129    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863132    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863135    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863138    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863141    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863144    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863147    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863150    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863152    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863155    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863158    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863160    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863163    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863165    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863168    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863171    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863173    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863176    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:28.871571 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863178    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863181    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863183    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863186    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863188    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863191    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863194    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863197    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863201    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863205    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863209    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863213    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863216    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863219    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863221    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863224    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863227    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863229    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863232    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:28.872083 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863234    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863237    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863239    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863242    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863244    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863247    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863249    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863252    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863255    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863258    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863260    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863263    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863265    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863268    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863270    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863273    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863276    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863278    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863281    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863284    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:28.872576 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863287    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863301    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863304    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863307    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863309    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863312    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863315    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863317    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863320    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863322    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863325    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863327    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863330    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863332    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863335    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863337    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863340    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863342    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863345    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863348    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:28.873066 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863350    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863357    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863360    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.863362    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.863947    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.870174    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.870189    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.870812    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.870820    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870874    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870879    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870882    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870885    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870887    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870890    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:28.873652 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870893    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870896    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870899    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870901    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870904    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870907    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870909    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870912    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870914    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870917    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870920    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870922    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870925    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870928    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870930    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870933    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870935    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870938    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870941    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870944    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:28.874029 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870946    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870949    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870952    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870955    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870958    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870961    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870963    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870966    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870968    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870971    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870973    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870976    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870978    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870981    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870983    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870986    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870989    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870991    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870994    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:28.874535 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870996    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.870999    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871002    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871006    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871009    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871013    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871016    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871018    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871021    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871024    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871027    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871029    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871032    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871034    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871037    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871039    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871043    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871046    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871049    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:28.875000 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871051    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871054    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871056    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871059    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871062    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871065    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871067    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871070    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871073    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871076    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871079    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871082    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871084    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871087    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871090    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871092    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871095    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871099    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871102    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:28.875527 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871105    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871108    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871111    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871115    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871117    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871120    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871122    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871125    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871128    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871130    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871134    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871136    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871139    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871143    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871147    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:28.875985 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871149    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.871154    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871260    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871264    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871267    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871270    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871273    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871275    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871278    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871281    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871284    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871286    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871289    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871304    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871308    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:28.876381 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871310    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871313    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871315    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871318    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871320    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871323    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871326    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871328    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871331    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871334    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871336    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871339    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871341    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871344    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871347    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871350    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871353    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871355    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871358    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871360    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:28.876757 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871363    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871365    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871368    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871370    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871373    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871375    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871378    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871380    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871383    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871385    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871388    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871390    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871393    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871395    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871398    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871400    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871403    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871405    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871408    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871410    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:28.877257 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871413    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871415    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871418    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871420    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871423    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871425    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871428    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871432    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871435    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871438    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871441    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871443    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871446    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871448    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871451    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871453    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871456    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871458    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871461    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:28.877770 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871463    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871466    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871468    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871471    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871473    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871476    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871478    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871481    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871483    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871486    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871488    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871491    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871493    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871495    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871498    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871501    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871505    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871508    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871511    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871513    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:28.878229 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871516    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871519    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871522    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871524    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871528    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871532    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:28.871535    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.871540    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.871633    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.873563    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.874263    2561 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.874364    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.874404    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:28.878893 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.876327    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:28.879872 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.879856    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:28.888349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.888330    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:28.890728 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.890713    2561 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:28.891967 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.891953    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:28.895774 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.895751    2561 fs.go:135] Filesystem UUIDs: map[1b5b4125-7a56-439a-aae7-062132b9ff47:/dev/nvme0n1p4 400d8fdb-ff93-4c01-be37-fb29e1727c2c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 16 18:40:28.895845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.895775    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:28.901498 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.901393    2561 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:28.900316614 +0000 UTC m=+0.255181243 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3079461 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec219ca9a46d389e785a7662a9dffa56 SystemUUID:ec219ca9-a46d-389e-785a-7662a9dffa56 BootID:765133e0-d8cc-4635-b15f-0a11f8e8137d Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e7:1e:18:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e7:1e:18:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:a0:9a:02:ad:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:28.901498 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.901492    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:28.901622 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.901556    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:28.904019 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.903993    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:28.904150 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.904021    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:28.904634 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.904624    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:28.904666 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.904636    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:28.904666 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.904643    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:28.905147 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.905138    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:28.906205 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.906195    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:28.909344 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.909327    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:28.910447 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.910434    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:28.910561 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.910551    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:28.910608 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.910586    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:28.910608 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.910600    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:28.910667 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.910616    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:28.910667 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.910624    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:28.910667 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.910632    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:28.910973 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.910959    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:28.911022 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.910983    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:28.912752 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.912733    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:28.914756 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.914742    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:28.914807 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.914772    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:28.916185 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916172    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:28.916218 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916194    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:28.916218 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916202    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:28.916218 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916209    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:28.916316 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916220    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:28.916316 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916233    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:28.916316 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916247    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:28.916316 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916252    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:28.916316 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916258    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:28.916316 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916263    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:28.916316 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916271    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:28.916316 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916276    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:28.916857 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916848    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:28.916857 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.916858    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:28.920289 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.920274    2561 server.go:1268] "Started kubelet"
Sep 16 18:40:28.920385 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.920324    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:28.920438 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.920396    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:28.920488 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.920444    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:28.921105 ip-10-0-128-29 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:28.922330 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.922309    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:28.924154 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.924020    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:28.924154 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.924113    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:28.924281 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.924153    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:28.924393 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.924374    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:28.924683 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.924660    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:28.928230 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.927461    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-29.ec2.internal.18d5e131537fec8b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-29.ec2.internal,UID:ip-10-0-128-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-29.ec2.internal,},FirstTimestamp:2026-09-16 18:40:28.920179851 +0000 UTC m=+0.275044480,LastTimestamp:2026-09-16 18:40:28.920179851 +0000 UTC m=+0.275044480,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-29.ec2.internal,}"
Sep 16 18:40:28.929155 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.929135    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:28.929570 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.929550    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:28.929725 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.929708    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:28.930062 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930044    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:28.930062 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930045    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:28.930195 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930072    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:28.930195 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930188    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:28.930195 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930196    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:28.930583 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930560    2561 factory.go:153] Registering CRI-O factory
Sep 16 18:40:28.930660 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930603    2561 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:28.930660 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930644    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:28.930660 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930650    2561 factory.go:55] Registering systemd factory
Sep 16 18:40:28.930660 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930655    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:28.930774 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930667    2561 factory.go:103] Registering Raw factory
Sep 16 18:40:28.930774 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930675    2561 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:28.930923 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.930914    2561 manager.go:349] Starting recovery of all containers
Sep 16 18:40:28.934187 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.934161    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:28.934327 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.934283    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:28.936196 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.936164    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:28.942131 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.942001    2561 manager.go:354] Recovery completed
Sep 16 18:40:28.946866 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.946851    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:28.952246 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.952223    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:28.952324 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.952258    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:28.952324 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.952268    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:28.952783 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.952768    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:28.952783 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.952778    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:28.952783 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.952781    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:28.952909 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.952798    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:28.953497 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.953484    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:28.953549 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.953508    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:28.953549 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:28.953519    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:28.954109 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.954041    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-29.ec2.internal.18d5e131556935e2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-29.ec2.internal,UID:ip-10-0-128-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-29.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-29.ec2.internal,},FirstTimestamp:2026-09-16 18:40:28.95224573 +0000 UTC m=+0.307110359,LastTimestamp:2026-09-16 18:40:28.95224573 +0000 UTC m=+0.307110359,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-29.ec2.internal,}"
Sep 16 18:40:28.963192 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.963135    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-29.ec2.internal.18d5e13155697875  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-29.ec2.internal,UID:ip-10-0-128-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-29.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-29.ec2.internal,},FirstTimestamp:2026-09-16 18:40:28.952262773 +0000 UTC m=+0.307127402,LastTimestamp:2026-09-16 18:40:28.952262773 +0000 UTC m=+0.307127402,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-29.ec2.internal,}"
Sep 16 18:40:28.974480 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.974420    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-29.ec2.internal.18d5e13155699eaf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-29.ec2.internal,UID:ip-10-0-128-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-29.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-29.ec2.internal,},FirstTimestamp:2026-09-16 18:40:28.952272559 +0000 UTC m=+0.307137188,LastTimestamp:2026-09-16 18:40:28.952272559 +0000 UTC m=+0.307137188,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-29.ec2.internal,}"
Sep 16 18:40:28.982266 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.982211    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-29.ec2.internal.18d5e131556935e2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-29.ec2.internal.18d5e131556935e2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-29.ec2.internal,UID:ip-10-0-128-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-29.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-29.ec2.internal,},FirstTimestamp:2026-09-16 18:40:28.95224573 +0000 UTC m=+0.307110359,LastTimestamp:2026-09-16 18:40:28.953497037 +0000 UTC m=+0.308361666,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-29.ec2.internal,}"
Sep 16 18:40:28.993483 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:28.993423    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-29.ec2.internal.18d5e13155697875\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-29.ec2.internal.18d5e13155697875  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-29.ec2.internal,UID:ip-10-0-128-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-29.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-29.ec2.internal,},FirstTimestamp:2026-09-16 18:40:28.952262773 +0000 UTC m=+0.307127402,LastTimestamp:2026-09-16 18:40:28.953512656 +0000 UTC m=+0.308377285,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-29.ec2.internal,}"
Sep 16 18:40:29.004210 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.004159    2561 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-29.ec2.internal.18d5e13155699eaf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-29.ec2.internal.18d5e13155699eaf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-29.ec2.internal,UID:ip-10-0-128-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-29.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-29.ec2.internal,},FirstTimestamp:2026-09-16 18:40:28.952272559 +0000 UTC m=+0.307137188,LastTimestamp:2026-09-16 18:40:28.953523144 +0000 UTC m=+0.308387773,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-29.ec2.internal,}"
Sep 16 18:40:29.030322 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.030299    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.080633 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.080615    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6hl9p"
Sep 16 18:40:29.091845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.091828    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6hl9p"
Sep 16 18:40:29.097257 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.097245    2561 policy_none.go:49] "None policy: Start"
Sep 16 18:40:29.097324 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.097262    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:29.097324 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.097272    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:29.099330 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.099315    2561 policy_none.go:47] "Start"
Sep 16 18:40:29.130721 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.130703    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.136661 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.136647    2561 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:29.136741 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.136691    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:29.136741 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.136703    2561 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:29.136973 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.136959    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:29.137039 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.136973    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:29.137093 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.137063    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:29.137171 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.137152    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:29.137171 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.137167    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:29.137719 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.137699    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:29.137803 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.137753    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.146747 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.146730    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:29.146815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.146772    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:29.146815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.146787    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:29.146883 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.146818    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:29.147307 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.147271    2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-29.ec2.internal\" not found" node="ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.148542 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.148527    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:29.237969 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.237947    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:29.238751 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.238737    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:29.238838 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.238767    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:29.238838 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.238783    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:29.238838 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.238807    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.246554 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.246539    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.246631 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.246562    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-29.ec2.internal\": node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.246901 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.246881    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"]
Sep 16 18:40:29.246964 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.246952    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:29.248175 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.248157    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:29.248236 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.248186    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:29.248236 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.248198    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:29.250541 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.250524    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:29.250698 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.250683    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.250774 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.250719    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:29.251248 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.251231    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:29.251248 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.251241    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:29.251391 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.251259    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:29.251391 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.251264    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:29.251391 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.251274    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:29.251391 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.251317    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:29.253525 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.253510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.253603 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.253533    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:29.254204 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.254187    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:29.254282 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.254217    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:29.254282 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.254232    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:29.257428 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.257407    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.269785 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.269768    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-29.ec2.internal\" not found" node="ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.273523 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.273510    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-29.ec2.internal\" not found" node="ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.331470 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.331448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.331528 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.331472    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.331528 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.331490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdf7e40390d26689763c696c18fb8f21-config\") pod \"kube-apiserver-proxy-ip-10-0-128-29.ec2.internal\" (UID: \"fdf7e40390d26689763c696c18fb8f21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.358247 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.358207    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.432006 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.431979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.432097 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.431943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.432097 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.432049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.432097 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.432068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdf7e40390d26689763c696c18fb8f21-config\") pod \"kube-apiserver-proxy-ip-10-0-128-29.ec2.internal\" (UID: \"fdf7e40390d26689763c696c18fb8f21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.432248 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.432103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdf7e40390d26689763c696c18fb8f21-config\") pod \"kube-apiserver-proxy-ip-10-0-128-29.ec2.internal\" (UID: \"fdf7e40390d26689763c696c18fb8f21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.432248 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.432141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.459084 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.459056    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.559570 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.559530    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.571693 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.571676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.577587 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.577567    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:29.660389 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.660335    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.760830 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.760799    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.861493 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.861472    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.873844 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.873815    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:29.873973 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.873952    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:29.962272 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:29.961515    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:29.991155 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:29.991136    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:30.061936 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:30.061913    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:30.095290 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.095255    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:29 +0000 UTC" deadline="2028-03-28 21:59:14.70427208 +0000 UTC"
Sep 16 18:40:30.095399 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.095292    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13419h18m44.608988324s"
Sep 16 18:40:30.140967 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.140949    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:30.142221 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.142207    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:30.142299 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.142239    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:30.142299 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.142250    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:30.162345 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:30.162323    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:30.196436 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:30.196402    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf7e40390d26689763c696c18fb8f21.slice/crio-cf69bb2dd4f12aaa4a23d2862e5ed83476c636358ef15f7be2330a5987eb3017 WatchSource:0}: Error finding container cf69bb2dd4f12aaa4a23d2862e5ed83476c636358ef15f7be2330a5987eb3017: Status 404 returned error can't find the container with id cf69bb2dd4f12aaa4a23d2862e5ed83476c636358ef15f7be2330a5987eb3017
Sep 16 18:40:30.196689 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:30.196670    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42daa9f52c2b3824ddc9a288af86477f.slice/crio-d5a2e9f9fd4545d830b7815c29663fc7951ba8a743c83da3cc2e60f41b4c548e WatchSource:0}: Error finding container d5a2e9f9fd4545d830b7815c29663fc7951ba8a743c83da3cc2e60f41b4c548e: Status 404 returned error can't find the container with id d5a2e9f9fd4545d830b7815c29663fc7951ba8a743c83da3cc2e60f41b4c548e
Sep 16 18:40:30.200904 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.200886    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:30.263556 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:30.263503    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:30.364344 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:30.364317    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:30.466205 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:30.466175    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 16 18:40:30.506819 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.506797    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:30.518468 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.518419    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:30.530413 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.530390    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:30.546348 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.546330    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:30.547482 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.547455    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 16 18:40:30.558478 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.558456    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:30.913434 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.913357    2561 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:30.925651 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.925476    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:30.926024 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.925989    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gf5lr","openshift-network-operator/iptables-alerter-vcd4n","openshift-ovn-kubernetes/ovnkube-node-z7clt","kube-system/global-pull-secret-syncer-96kgg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb","openshift-dns/node-resolver-l8qxv","openshift-multus/multus-additional-cni-plugins-pp8w7","openshift-multus/network-metrics-daemon-dwzxx","openshift-network-diagnostics/network-check-target-n658c","kube-system/konnectivity-agent-vslg4","kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xhjgx","openshift-image-registry/node-ca-vxjwf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"]
Sep 16 18:40:30.928733 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.928707    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:30.930628 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.930439    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:30.932312 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.932291    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:30.932481 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.932460    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6f6rf\""
Sep 16 18:40:30.932552 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.932531    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:30.932745 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.932470    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:30.932883 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.932865    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:30.932964 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.932945    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:30.933065 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.933048    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:30.934911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.934892    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:30.936323 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.936301    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zv5rc\""
Sep 16 18:40:30.936453 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.936352    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:30.936694 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.936663    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:30.936798 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.936717    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:30.937105 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.937087    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:30.937197 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:30.937161    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:30.938825 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.938785    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:30.938943 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.938885    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x4cbm\""
Sep 16 18:40:30.938943 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.938904    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:30.938943 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.938794    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:30.938943 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.938929    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:30.939534 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.939517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:30.940754 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.940735    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:30.940754 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.940743    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:30.941736 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmq6x\" (UniqueName: \"kubernetes.io/projected/b84db093-de67-44ec-ade5-7a4a84648e0d-kube-api-access-bmq6x\") pod \"node-ca-vxjwf\" (UID: \"b84db093-de67-44ec-ade5-7a4a84648e0d\") " pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:30.941814 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e2a384ac-d60b-4bbd-872d-dc42760a3455-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:30.941814 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgrt\" (UniqueName: \"kubernetes.io/projected/e2a384ac-d60b-4bbd-872d-dc42760a3455-kube-api-access-fkgrt\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:30.941911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b84db093-de67-44ec-ade5-7a4a84648e0d-host\") pod \"node-ca-vxjwf\" (UID: \"b84db093-de67-44ec-ade5-7a4a84648e0d\") " pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:30.941911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941837    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b84db093-de67-44ec-ade5-7a4a84648e0d-serviceca\") pod \"node-ca-vxjwf\" (UID: \"b84db093-de67-44ec-ade5-7a4a84648e0d\") " pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:30.941911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2a384ac-d60b-4bbd-872d-dc42760a3455-system-cni-dir\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:30.941911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2a384ac-d60b-4bbd-872d-dc42760a3455-cnibin\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:30.942046 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2a384ac-d60b-4bbd-872d-dc42760a3455-cni-binary-copy\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:30.942046 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2a384ac-d60b-4bbd-872d-dc42760a3455-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:30.942046 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.941992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e2a384ac-d60b-4bbd-872d-dc42760a3455-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:30.942487 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.942459    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:30.942688 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.942669    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lnhvl\""
Sep 16 18:40:30.943348 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.943107    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:30.943436 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.943363    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:30.943602 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.943534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:30.945933 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.945901    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:30.946031 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:30.945973    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:30.946385 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.946358    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:30.946847 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.946814    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m6kgg\""
Sep 16 18:40:30.946948 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.946850    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:30.949397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.948157    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:30.949397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.948462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:30.951113 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.951082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:30.951766 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.951745    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:30.951899 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.951852    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-525qs\""
Sep 16 18:40:30.953780 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.953761    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:30.954313 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.954291    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:30.955383 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.955348    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:30.955484 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.955426    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q4qr9\""
Sep 16 18:40:30.955484 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.955479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:30.956630 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.956609    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:30.956724 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.956707    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:30.956815 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:30.956762    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:30.956962 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.956940    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c8dx8\""
Sep 16 18:40:30.957016 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.956960    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:30.958878 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.958861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:30.962705 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.962686    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:30.962705 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.962697    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:30.962859 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:30.962732    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4vt5b\""
Sep 16 18:40:31.031403 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.031383    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:31.043162 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-sys-fs\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.043254 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-systemd\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.043254 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-ovn-node-metrics-cert\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.043254 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-run-k8s-cni-cncf-io\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.043254 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-var-lib-kubelet\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.043448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl277\" (UniqueName: \"kubernetes.io/projected/43c13f0d-db2b-46e7-87b6-50fa4666087a-kube-api-access-fl277\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.043448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:31.043448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043330    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-sysconfig\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.043448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-host\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.043448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-systemd-units\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.043448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-daemon-config\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.043448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2a384ac-d60b-4bbd-872d-dc42760a3455-system-cni-dir\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.043448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e2a384ac-d60b-4bbd-872d-dc42760a3455-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-sys\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pcb\" (UniqueName: \"kubernetes.io/projected/0d4e3ca8-8af4-492f-8d27-68d663b861fe-kube-api-access-29pcb\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-var-lib-openvswitch\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-node-log\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sptmf\" (UniqueName: \"kubernetes.io/projected/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-kube-api-access-sptmf\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043565    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d882ea-dbac-48fe-9406-ad43a64e7170-tmp-dir\") pod \"node-resolver-l8qxv\" (UID: \"73d882ea-dbac-48fe-9406-ad43a64e7170\") " pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043597    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-lib-modules\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-env-overrides\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43c13f0d-db2b-46e7-87b6-50fa4666087a-cni-binary-copy\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13da0f1f-abdc-4b10-a33d-25aff53b9c84-agent-certs\") pod \"konnectivity-agent-vslg4\" (UID: \"13da0f1f-abdc-4b10-a33d-25aff53b9c84\") " pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-socket-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-etc-selinux\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq6x\" (UniqueName: \"kubernetes.io/projected/b84db093-de67-44ec-ade5-7a4a84648e0d-kube-api-access-bmq6x\") pod \"node-ca-vxjwf\" (UID: \"b84db093-de67-44ec-ade5-7a4a84648e0d\") " pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-kubernetes\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-sysctl-conf\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.043815 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-run-openvswitch\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043810    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-ovnkube-script-lib\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/012ca71e-d6e0-45af-a910-83d05cc0dd6b-dbus\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73d882ea-dbac-48fe-9406-ad43a64e7170-hosts-file\") pod \"node-resolver-l8qxv\" (UID: \"73d882ea-dbac-48fe-9406-ad43a64e7170\") " pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b84db093-de67-44ec-ade5-7a4a84648e0d-host\") pod \"node-ca-vxjwf\" (UID: \"b84db093-de67-44ec-ade5-7a4a84648e0d\") " pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-sysctl-d\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-kubelet\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-run-netns\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.043980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-socket-dir-parent\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-device-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jnbf\" (UniqueName: \"kubernetes.io/projected/80becee4-0e8e-437c-955b-e9cb44506e22-kube-api-access-4jnbf\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2a384ac-d60b-4bbd-872d-dc42760a3455-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044084    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-modprobe-d\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-run\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-run-systemd\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.044523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfq6\" (UniqueName: \"kubernetes.io/projected/73d882ea-dbac-48fe-9406-ad43a64e7170-kube-api-access-srfq6\") pod \"node-resolver-l8qxv\" (UID: \"73d882ea-dbac-48fe-9406-ad43a64e7170\") " pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgrt\" (UniqueName: \"kubernetes.io/projected/e2a384ac-d60b-4bbd-872d-dc42760a3455-kube-api-access-fkgrt\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b84db093-de67-44ec-ade5-7a4a84648e0d-serviceca\") pod \"node-ca-vxjwf\" (UID: \"b84db093-de67-44ec-ade5-7a4a84648e0d\") " pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-cni-dir\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-var-lib-cni-multus\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13da0f1f-abdc-4b10-a33d-25aff53b9c84-konnectivity-ca\") pod \"konnectivity-agent-vslg4\" (UID: \"13da0f1f-abdc-4b10-a33d-25aff53b9c84\") " pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hmn\" (UniqueName: \"kubernetes.io/projected/54562fc0-f30a-4a41-bef2-dd2b3d51924b-kube-api-access-69hmn\") pod \"iptables-alerter-vcd4n\" (UID: \"54562fc0-f30a-4a41-bef2-dd2b3d51924b\") " pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-var-lib-kubelet\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-var-lib-cni-bin\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-registration-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-cni-netd\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54562fc0-f30a-4a41-bef2-dd2b3d51924b-iptables-alerter-script\") pod \"iptables-alerter-vcd4n\" (UID: \"54562fc0-f30a-4a41-bef2-dd2b3d51924b\") " pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044529    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044559    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2a384ac-d60b-4bbd-872d-dc42760a3455-cnibin\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-hostroot\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.045127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-conf-dir\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54562fc0-f30a-4a41-bef2-dd2b3d51924b-host-slash\") pod \"iptables-alerter-vcd4n\" (UID: \"54562fc0-f30a-4a41-bef2-dd2b3d51924b\") " pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-slash\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-log-socket\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/012ca71e-d6e0-45af-a910-83d05cc0dd6b-kubelet-config\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-run-netns\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-system-cni-dir\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2a384ac-d60b-4bbd-872d-dc42760a3455-cni-binary-copy\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-etc-openvswitch\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-ovnkube-config\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-run-multus-certs\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qjp\" (UniqueName: \"kubernetes.io/projected/d40797dc-8666-4bf3-9eca-3436c9271385-kube-api-access-l6qjp\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e2a384ac-d60b-4bbd-872d-dc42760a3455-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-tuned\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-run-ovn\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-cnibin\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.045879 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.044991    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d4e3ca8-8af4-492f-8d27-68d663b861fe-tmp\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.045007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-cni-bin\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.045024    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.045041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-etc-kubernetes\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.045214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2a384ac-d60b-4bbd-872d-dc42760a3455-system-cni-dir\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.045736    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e2a384ac-d60b-4bbd-872d-dc42760a3455-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.046005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2a384ac-d60b-4bbd-872d-dc42760a3455-cnibin\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.046336    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b84db093-de67-44ec-ade5-7a4a84648e0d-host\") pod \"node-ca-vxjwf\" (UID: \"b84db093-de67-44ec-ade5-7a4a84648e0d\") " pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.046438    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2a384ac-d60b-4bbd-872d-dc42760a3455-cni-binary-copy\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.046542    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2a384ac-d60b-4bbd-872d-dc42760a3455-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.046712 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.046570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b84db093-de67-44ec-ade5-7a4a84648e0d-serviceca\") pod \"node-ca-vxjwf\" (UID: \"b84db093-de67-44ec-ade5-7a4a84648e0d\") " pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:31.047177 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.047096    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e2a384ac-d60b-4bbd-872d-dc42760a3455-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.056564 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.056540    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nlwts"
Sep 16 18:40:31.059474 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.059455    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:31.063873 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.063853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgrt\" (UniqueName: \"kubernetes.io/projected/e2a384ac-d60b-4bbd-872d-dc42760a3455-kube-api-access-fkgrt\") pod \"multus-additional-cni-plugins-pp8w7\" (UID: \"e2a384ac-d60b-4bbd-872d-dc42760a3455\") " pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.064619 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.064584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq6x\" (UniqueName: \"kubernetes.io/projected/b84db093-de67-44ec-ade5-7a4a84648e0d-kube-api-access-bmq6x\") pod \"node-ca-vxjwf\" (UID: \"b84db093-de67-44ec-ade5-7a4a84648e0d\") " pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:31.065888 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.065836    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nlwts"
Sep 16 18:40:31.145406 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54562fc0-f30a-4a41-bef2-dd2b3d51924b-iptables-alerter-script\") pod \"iptables-alerter-vcd4n\" (UID: \"54562fc0-f30a-4a41-bef2-dd2b3d51924b\") " pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.145544 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:31.145544 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-hostroot\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.145544 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-conf-dir\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.145544 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54562fc0-f30a-4a41-bef2-dd2b3d51924b-host-slash\") pod \"iptables-alerter-vcd4n\" (UID: \"54562fc0-f30a-4a41-bef2-dd2b3d51924b\") " pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.145544 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-slash\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-log-socket\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/012ca71e-d6e0-45af-a910-83d05cc0dd6b-kubelet-config\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-run-netns\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-hostroot\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-conf-dir\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.145691    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145721    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54562fc0-f30a-4a41-bef2-dd2b3d51924b-host-slash\") pod \"iptables-alerter-vcd4n\" (UID: \"54562fc0-f30a-4a41-bef2-dd2b3d51924b\") " pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-slash\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.145783    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs podName:80becee4-0e8e-437c-955b-e9cb44506e22 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.645759362 +0000 UTC m=+3.000624006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs") pod "network-metrics-daemon-dwzxx" (UID: "80becee4-0e8e-437c-955b-e9cb44506e22") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-log-socket\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.145824 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-system-cni-dir\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-etc-openvswitch\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/012ca71e-d6e0-45af-a910-83d05cc0dd6b-kubelet-config\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-ovnkube-config\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-etc-openvswitch\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-run-multus-certs\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qjp\" (UniqueName: \"kubernetes.io/projected/d40797dc-8666-4bf3-9eca-3436c9271385-kube-api-access-l6qjp\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145917    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-run-netns\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145945    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-tuned\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-run-ovn\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-cnibin\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-cnibin\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-run-ovn\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d4e3ca8-8af4-492f-8d27-68d663b861fe-tmp\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-cni-bin\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:31.146459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-etc-kubernetes\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-sys-fs\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146297    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-systemd\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-ovn-node-metrics-cert\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-cni-bin\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-run-k8s-cni-cncf-io\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146398    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-var-lib-kubelet\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146430    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl277\" (UniqueName: \"kubernetes.io/projected/43c13f0d-db2b-46e7-87b6-50fa4666087a-kube-api-access-fl277\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-ovnkube-config\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-system-cni-dir\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-sysconfig\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-host\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-run-k8s-cni-cncf-io\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-systemd-units\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-daemon-config\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-systemd-units\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146608    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-sys\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.147397 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29pcb\" (UniqueName: \"kubernetes.io/projected/0d4e3ca8-8af4-492f-8d27-68d663b861fe-kube-api-access-29pcb\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-var-lib-openvswitch\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-sysconfig\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-node-log\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-host\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sptmf\" (UniqueName: \"kubernetes.io/projected/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-kube-api-access-sptmf\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d882ea-dbac-48fe-9406-ad43a64e7170-tmp-dir\") pod \"node-resolver-l8qxv\" (UID: \"73d882ea-dbac-48fe-9406-ad43a64e7170\") " pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.145913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-run-multus-certs\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-lib-modules\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146811    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-env-overrides\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43c13f0d-db2b-46e7-87b6-50fa4666087a-cni-binary-copy\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146865    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13da0f1f-abdc-4b10-a33d-25aff53b9c84-agent-certs\") pod \"konnectivity-agent-vslg4\" (UID: \"13da0f1f-abdc-4b10-a33d-25aff53b9c84\") " pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-socket-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-etc-selinux\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146954    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-kubernetes\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-sysctl-conf\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-run-openvswitch\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.148349 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-ovnkube-script-lib\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/012ca71e-d6e0-45af-a910-83d05cc0dd6b-dbus\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73d882ea-dbac-48fe-9406-ad43a64e7170-hosts-file\") pod \"node-resolver-l8qxv\" (UID: \"73d882ea-dbac-48fe-9406-ad43a64e7170\") " pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-sysctl-d\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147133    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54562fc0-f30a-4a41-bef2-dd2b3d51924b-iptables-alerter-script\") pod \"iptables-alerter-vcd4n\" (UID: \"54562fc0-f30a-4a41-bef2-dd2b3d51924b\") " pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-etc-kubernetes\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.147195    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-kubernetes\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-socket-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.147251    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret podName:012ca71e-d6e0-45af-a910-83d05cc0dd6b nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.64723498 +0000 UTC m=+3.002099627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret") pod "global-pull-secret-syncer-96kgg" (UID: "012ca71e-d6e0-45af-a910-83d05cc0dd6b") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147256    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-lib-modules\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-kubelet\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.146635    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-var-lib-kubelet\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d882ea-dbac-48fe-9406-ad43a64e7170-tmp-dir\") pod \"node-resolver-l8qxv\" (UID: \"73d882ea-dbac-48fe-9406-ad43a64e7170\") " pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147427    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-etc-selinux\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/012ca71e-d6e0-45af-a910-83d05cc0dd6b-dbus\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-sysctl-conf\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.149237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-env-overrides\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-run-openvswitch\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-var-lib-openvswitch\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.147943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43c13f0d-db2b-46e7-87b6-50fa4666087a-cni-binary-copy\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-run-netns\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-daemon-config\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-socket-dir-parent\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-device-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148137    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-systemd\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jnbf\" (UniqueName: \"kubernetes.io/projected/80becee4-0e8e-437c-955b-e9cb44506e22-kube-api-access-4jnbf\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-modprobe-d\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-sys-fs\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148234    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-sysctl-d\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-run\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-node-log\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-modprobe-d\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.150125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-run-systemd\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-run-systemd\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfq6\" (UniqueName: \"kubernetes.io/projected/73d882ea-dbac-48fe-9406-ad43a64e7170-kube-api-access-srfq6\") pod \"node-resolver-l8qxv\" (UID: \"73d882ea-dbac-48fe-9406-ad43a64e7170\") " pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-cni-dir\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-var-lib-cni-multus\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-socket-dir-parent\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-run\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13da0f1f-abdc-4b10-a33d-25aff53b9c84-konnectivity-ca\") pod \"konnectivity-agent-vslg4\" (UID: \"13da0f1f-abdc-4b10-a33d-25aff53b9c84\") " pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73d882ea-dbac-48fe-9406-ad43a64e7170-hosts-file\") pod \"node-resolver-l8qxv\" (UID: \"73d882ea-dbac-48fe-9406-ad43a64e7170\") " pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hmn\" (UniqueName: \"kubernetes.io/projected/54562fc0-f30a-4a41-bef2-dd2b3d51924b-kube-api-access-69hmn\") pod \"iptables-alerter-vcd4n\" (UID: \"54562fc0-f30a-4a41-bef2-dd2b3d51924b\") " pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-var-lib-kubelet\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-var-lib-cni-bin\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-registration-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-cni-netd\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-multus-cni-dir\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.151029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148951    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-kubelet\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-var-lib-cni-multus\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.148995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43c13f0d-db2b-46e7-87b6-50fa4666087a-host-var-lib-cni-bin\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149020    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-var-lib-kubelet\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149131    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-cni-netd\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-ovnkube-script-lib\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-host-run-netns\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149322    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-registration-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d40797dc-8666-4bf3-9eca-3436c9271385-device-dir\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d4e3ca8-8af4-492f-8d27-68d663b861fe-sys\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149679    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13da0f1f-abdc-4b10-a33d-25aff53b9c84-konnectivity-ca\") pod \"konnectivity-agent-vslg4\" (UID: \"13da0f1f-abdc-4b10-a33d-25aff53b9c84\") " pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149842    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d4e3ca8-8af4-492f-8d27-68d663b861fe-etc-tuned\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.149859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d4e3ca8-8af4-492f-8d27-68d663b861fe-tmp\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.151845 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.150659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13da0f1f-abdc-4b10-a33d-25aff53b9c84-agent-certs\") pod \"konnectivity-agent-vslg4\" (UID: \"13da0f1f-abdc-4b10-a33d-25aff53b9c84\") " pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:31.152545 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.152157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-ovn-node-metrics-cert\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.153023 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.152965    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal" event={"ID":"fdf7e40390d26689763c696c18fb8f21","Type":"ContainerStarted","Data":"cf69bb2dd4f12aaa4a23d2862e5ed83476c636358ef15f7be2330a5987eb3017"}
Sep 16 18:40:31.154658 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.154634    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" event={"ID":"42daa9f52c2b3824ddc9a288af86477f","Type":"ContainerStarted","Data":"d5a2e9f9fd4545d830b7815c29663fc7951ba8a743c83da3cc2e60f41b4c548e"}
Sep 16 18:40:31.167801 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.167731    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:31.167801 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.167756    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:31.167801 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.167772    2561 projected.go:196] Error preparing data for projected volume kube-api-access-rvt6z for pod openshift-network-diagnostics/network-check-target-n658c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:31.168009 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.167828    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z podName:6b125d04-b3b2-42a1-8d4e-99c6e5242240 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.66781364 +0000 UTC m=+3.022678276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rvt6z" (UniqueName: "kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z") pod "network-check-target-n658c" (UID: "6b125d04-b3b2-42a1-8d4e-99c6e5242240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:31.170368 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.170230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qjp\" (UniqueName: \"kubernetes.io/projected/d40797dc-8666-4bf3-9eca-3436c9271385-kube-api-access-l6qjp\") pod \"aws-ebs-csi-driver-node-gvtdb\" (UID: \"d40797dc-8666-4bf3-9eca-3436c9271385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.172906 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.172886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl277\" (UniqueName: \"kubernetes.io/projected/43c13f0d-db2b-46e7-87b6-50fa4666087a-kube-api-access-fl277\") pod \"multus-gf5lr\" (UID: \"43c13f0d-db2b-46e7-87b6-50fa4666087a\") " pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.173006 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.172974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jnbf\" (UniqueName: \"kubernetes.io/projected/80becee4-0e8e-437c-955b-e9cb44506e22-kube-api-access-4jnbf\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:31.173497 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.173475    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pcb\" (UniqueName: \"kubernetes.io/projected/0d4e3ca8-8af4-492f-8d27-68d663b861fe-kube-api-access-29pcb\") pod \"tuned-xhjgx\" (UID: \"0d4e3ca8-8af4-492f-8d27-68d663b861fe\") " pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.173599 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.173489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sptmf\" (UniqueName: \"kubernetes.io/projected/3dd8fb7f-cee2-4625-af27-51acec6ecdc6-kube-api-access-sptmf\") pod \"ovnkube-node-z7clt\" (UID: \"3dd8fb7f-cee2-4625-af27-51acec6ecdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.174442 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.174420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hmn\" (UniqueName: \"kubernetes.io/projected/54562fc0-f30a-4a41-bef2-dd2b3d51924b-kube-api-access-69hmn\") pod \"iptables-alerter-vcd4n\" (UID: \"54562fc0-f30a-4a41-bef2-dd2b3d51924b\") " pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.175511 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.175487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfq6\" (UniqueName: \"kubernetes.io/projected/73d882ea-dbac-48fe-9406-ad43a64e7170-kube-api-access-srfq6\") pod \"node-resolver-l8qxv\" (UID: \"73d882ea-dbac-48fe-9406-ad43a64e7170\") " pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.243948 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.243925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pp8w7"
Sep 16 18:40:31.252975 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.252948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vxjwf"
Sep 16 18:40:31.263914 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.263891    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:31.270300 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.270281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb"
Sep 16 18:40:31.278259 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.278239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l8qxv"
Sep 16 18:40:31.287276 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.287258    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gf5lr"
Sep 16 18:40:31.296641 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.296623    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vcd4n"
Sep 16 18:40:31.305482 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.305460    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:31.312327 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.312294    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhjgx"
Sep 16 18:40:31.403810 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.403784    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:31.652733 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.652662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:31.652733 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.652713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:31.652935 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.652841    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:31.652935 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.652843    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:31.652935 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.652897    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs podName:80becee4-0e8e-437c-955b-e9cb44506e22 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:32.652882029 +0000 UTC m=+4.007746646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs") pod "network-metrics-daemon-dwzxx" (UID: "80becee4-0e8e-437c-955b-e9cb44506e22") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:31.652935 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.652912    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret podName:012ca71e-d6e0-45af-a910-83d05cc0dd6b nodeName:}" failed. No retries permitted until 2026-09-16 18:40:32.65290476 +0000 UTC m=+4.007769382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret") pod "global-pull-secret-syncer-96kgg" (UID: "012ca71e-d6e0-45af-a910-83d05cc0dd6b") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:31.753439 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:31.753405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:31.753586 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.753527    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:31.753586 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.753559    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:31.753586 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.753568    2561 projected.go:196] Error preparing data for projected volume kube-api-access-rvt6z for pod openshift-network-diagnostics/network-check-target-n658c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:31.753719 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:31.753621    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z podName:6b125d04-b3b2-42a1-8d4e-99c6e5242240 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:32.753605015 +0000 UTC m=+4.108469646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvt6z" (UniqueName: "kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z") pod "network-check-target-n658c" (UID: "6b125d04-b3b2-42a1-8d4e-99c6e5242240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:32.070727 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:32.070627    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:31 +0000 UTC" deadline="2028-02-18 19:52:01.859199331 +0000 UTC"
Sep 16 18:40:32.070727 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:32.070678    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12481h11m29.788527021s"
Sep 16 18:40:32.147825 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:32.147774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:32.147942 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:32.147874    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:32.197093 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:32.197060    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a384ac_d60b_4bbd_872d_dc42760a3455.slice/crio-cfefd7f5626cf9cb11f81bac68d30cfbfaebb2cf7dbd13ec4212e76bf1d17cfb WatchSource:0}: Error finding container cfefd7f5626cf9cb11f81bac68d30cfbfaebb2cf7dbd13ec4212e76bf1d17cfb: Status 404 returned error can't find the container with id cfefd7f5626cf9cb11f81bac68d30cfbfaebb2cf7dbd13ec4212e76bf1d17cfb
Sep 16 18:40:32.198307 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:32.198229    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40797dc_8666_4bf3_9eca_3436c9271385.slice/crio-4f779713405ff707b591ae5778f6edc01bfd740d2a993acc31662253c953c870 WatchSource:0}: Error finding container 4f779713405ff707b591ae5778f6edc01bfd740d2a993acc31662253c953c870: Status 404 returned error can't find the container with id 4f779713405ff707b591ae5778f6edc01bfd740d2a993acc31662253c953c870
Sep 16 18:40:32.199803 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:32.199749    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d882ea_dbac_48fe_9406_ad43a64e7170.slice/crio-dd820a6acaf73d901002c49af8a561f9339e7ea96ef62ff8c517e4e8c62c2ea3 WatchSource:0}: Error finding container dd820a6acaf73d901002c49af8a561f9339e7ea96ef62ff8c517e4e8c62c2ea3: Status 404 returned error can't find the container with id dd820a6acaf73d901002c49af8a561f9339e7ea96ef62ff8c517e4e8c62c2ea3
Sep 16 18:40:32.202959 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:32.202934    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84db093_de67_44ec_ade5_7a4a84648e0d.slice/crio-5119b9cbb471657963e99463d02ea33d1f4bca041717d848f94af2bbc1cef2fb WatchSource:0}: Error finding container 5119b9cbb471657963e99463d02ea33d1f4bca041717d848f94af2bbc1cef2fb: Status 404 returned error can't find the container with id 5119b9cbb471657963e99463d02ea33d1f4bca041717d848f94af2bbc1cef2fb
Sep 16 18:40:32.204452 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:32.204404    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c13f0d_db2b_46e7_87b6_50fa4666087a.slice/crio-3ab7ba02805bc5665e07aca8f4444e4b92d7ebbebf7ca873b1a19b51eefb6787 WatchSource:0}: Error finding container 3ab7ba02805bc5665e07aca8f4444e4b92d7ebbebf7ca873b1a19b51eefb6787: Status 404 returned error can't find the container with id 3ab7ba02805bc5665e07aca8f4444e4b92d7ebbebf7ca873b1a19b51eefb6787
Sep 16 18:40:32.205997 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:40:32.205972    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13da0f1f_abdc_4b10_a33d_25aff53b9c84.slice/crio-f0892a0b0b5691abdbab1c640d2ac6a84dc104101ac6fbda732df7ec80328010 WatchSource:0}: Error finding container f0892a0b0b5691abdbab1c640d2ac6a84dc104101ac6fbda732df7ec80328010: Status 404 returned error can't find the container with id f0892a0b0b5691abdbab1c640d2ac6a84dc104101ac6fbda732df7ec80328010
Sep 16 18:40:32.662278 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:32.662239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:32.662463 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:32.662341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:32.662547 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:32.662478    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:32.662612 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:32.662558    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs podName:80becee4-0e8e-437c-955b-e9cb44506e22 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.662536253 +0000 UTC m=+6.017400910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs") pod "network-metrics-daemon-dwzxx" (UID: "80becee4-0e8e-437c-955b-e9cb44506e22") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:32.662612 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:32.662572    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:32.662731 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:32.662648    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret podName:012ca71e-d6e0-45af-a910-83d05cc0dd6b nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.662628835 +0000 UTC m=+6.017493473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret") pod "global-pull-secret-syncer-96kgg" (UID: "012ca71e-d6e0-45af-a910-83d05cc0dd6b") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:32.763214 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:32.763033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:32.763214 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:32.763209    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:32.763452 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:32.763255    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:32.763452 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:32.763269    2561 projected.go:196] Error preparing data for projected volume kube-api-access-rvt6z for pod openshift-network-diagnostics/network-check-target-n658c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:32.763452 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:32.763324    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z podName:6b125d04-b3b2-42a1-8d4e-99c6e5242240 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.763307544 +0000 UTC m=+6.118172168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvt6z" (UniqueName: "kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z") pod "network-check-target-n658c" (UID: "6b125d04-b3b2-42a1-8d4e-99c6e5242240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:33.071588 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.071463    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:31 +0000 UTC" deadline="2028-06-16 12:25:25.673099202 +0000 UTC"
Sep 16 18:40:33.071588 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.071522    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15329h44m52.60158198s"
Sep 16 18:40:33.154493 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.149465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:33.154493 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:33.149599    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:33.154493 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.149917    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:33.154493 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:33.150011    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:33.180153 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.180090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pp8w7" event={"ID":"e2a384ac-d60b-4bbd-872d-dc42760a3455","Type":"ContainerStarted","Data":"cfefd7f5626cf9cb11f81bac68d30cfbfaebb2cf7dbd13ec4212e76bf1d17cfb"}
Sep 16 18:40:33.183226 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.183193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal" event={"ID":"fdf7e40390d26689763c696c18fb8f21","Type":"ContainerStarted","Data":"77b4005509e824732b1be1c3b7905e752ae82699877c9a79c6d72b9bd5c3af87"}
Sep 16 18:40:33.197241 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.197213    2561 generic.go:356] "Generic (PLEG): container finished" podID="42daa9f52c2b3824ddc9a288af86477f" containerID="cb689426af94deaf83fa4887d865380f3ba90e1ec08b9855af62d61f9920ba80" exitCode=0
Sep 16 18:40:33.197337 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.197293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" event={"ID":"42daa9f52c2b3824ddc9a288af86477f","Type":"ContainerDied","Data":"cb689426af94deaf83fa4887d865380f3ba90e1ec08b9855af62d61f9920ba80"}
Sep 16 18:40:33.205090 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.205031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vcd4n" event={"ID":"54562fc0-f30a-4a41-bef2-dd2b3d51924b","Type":"ContainerStarted","Data":"96f7f4ad2326097046f507e4b82ce33df44ecd4e7e2ce471f38c88a76fe3b8f8"}
Sep 16 18:40:33.216567 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.216540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vslg4" event={"ID":"13da0f1f-abdc-4b10-a33d-25aff53b9c84","Type":"ContainerStarted","Data":"f0892a0b0b5691abdbab1c640d2ac6a84dc104101ac6fbda732df7ec80328010"}
Sep 16 18:40:33.222002 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.221968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l8qxv" event={"ID":"73d882ea-dbac-48fe-9406-ad43a64e7170","Type":"ContainerStarted","Data":"dd820a6acaf73d901002c49af8a561f9339e7ea96ef62ff8c517e4e8c62c2ea3"}
Sep 16 18:40:33.224711 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.224686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" event={"ID":"3dd8fb7f-cee2-4625-af27-51acec6ecdc6","Type":"ContainerStarted","Data":"018051ee5aa4fb8d90e7f5e596d49b071f5d19678703381e1b6107114d673b14"}
Sep 16 18:40:33.233551 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.233525    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhjgx" event={"ID":"0d4e3ca8-8af4-492f-8d27-68d663b861fe","Type":"ContainerStarted","Data":"d177da87d625a28a5f5e14be09a85a16d85082e4dd026712f8f4b13d3bc3fe51"}
Sep 16 18:40:33.245823 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.245796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gf5lr" event={"ID":"43c13f0d-db2b-46e7-87b6-50fa4666087a","Type":"ContainerStarted","Data":"3ab7ba02805bc5665e07aca8f4444e4b92d7ebbebf7ca873b1a19b51eefb6787"}
Sep 16 18:40:33.253806 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.252721    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal" podStartSLOduration=2.252703051 podStartE2EDuration="2.252703051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:33.21899554 +0000 UTC m=+4.573860182" watchObservedRunningTime="2026-09-16 18:40:33.252703051 +0000 UTC m=+4.607567693"
Sep 16 18:40:33.254075 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.254047    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vxjwf" event={"ID":"b84db093-de67-44ec-ade5-7a4a84648e0d","Type":"ContainerStarted","Data":"5119b9cbb471657963e99463d02ea33d1f4bca041717d848f94af2bbc1cef2fb"}
Sep 16 18:40:33.286160 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:33.286131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb" event={"ID":"d40797dc-8666-4bf3-9eca-3436c9271385","Type":"ContainerStarted","Data":"4f779713405ff707b591ae5778f6edc01bfd740d2a993acc31662253c953c870"}
Sep 16 18:40:34.147798 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:34.147327    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:34.147798 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:34.147451    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:34.304534 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:34.304431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" event={"ID":"42daa9f52c2b3824ddc9a288af86477f","Type":"ContainerStarted","Data":"099f53b67e2cc2380227d3648ae43e352fbf46ec723396aff50ab73b3c55d6a2"}
Sep 16 18:40:34.677964 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:34.677879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:34.678127 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:34.677972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:34.678127 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:34.678113    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.678247 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:34.678178    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret podName:012ca71e-d6e0-45af-a910-83d05cc0dd6b nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.678157862 +0000 UTC m=+10.033022495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret") pod "global-pull-secret-syncer-96kgg" (UID: "012ca71e-d6e0-45af-a910-83d05cc0dd6b") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.678617 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:34.678593    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.678721 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:34.678654    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs podName:80becee4-0e8e-437c-955b-e9cb44506e22 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.678638421 +0000 UTC m=+10.033503055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs") pod "network-metrics-daemon-dwzxx" (UID: "80becee4-0e8e-437c-955b-e9cb44506e22") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.779136 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:34.779096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:34.779348 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:34.779315    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:34.779458 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:34.779354    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:34.779458 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:34.779367    2561 projected.go:196] Error preparing data for projected volume kube-api-access-rvt6z for pod openshift-network-diagnostics/network-check-target-n658c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:34.779458 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:34.779422    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z podName:6b125d04-b3b2-42a1-8d4e-99c6e5242240 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:38.779405041 +0000 UTC m=+10.134269662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvt6z" (UniqueName: "kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z") pod "network-check-target-n658c" (UID: "6b125d04-b3b2-42a1-8d4e-99c6e5242240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:35.148528 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:35.148450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:35.148982 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:35.148608    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:35.148982 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:35.148663    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:35.148982 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:35.148787    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:36.147589 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:36.147053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:36.147589 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:36.147187    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:37.148755 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:37.148719    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:37.149261 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:37.148845    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:37.149386 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:37.149339    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:37.149476 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:37.149451    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:38.147260 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:38.146959    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:38.147431 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:38.147346    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:38.713218 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:38.713180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:38.713636 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:38.713256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:38.713636 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:38.713378    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:38.713636 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:38.713409    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:38.713636 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:38.713466    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs podName:80becee4-0e8e-437c-955b-e9cb44506e22 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:46.713442655 +0000 UTC m=+18.068307298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs") pod "network-metrics-daemon-dwzxx" (UID: "80becee4-0e8e-437c-955b-e9cb44506e22") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:38.713636 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:38.713488    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret podName:012ca71e-d6e0-45af-a910-83d05cc0dd6b nodeName:}" failed. No retries permitted until 2026-09-16 18:40:46.713477615 +0000 UTC m=+18.068342239 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret") pod "global-pull-secret-syncer-96kgg" (UID: "012ca71e-d6e0-45af-a910-83d05cc0dd6b") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:38.814608 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:38.814574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:38.814776 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:38.814756    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:38.814836 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:38.814779    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:38.814836 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:38.814792    2561 projected.go:196] Error preparing data for projected volume kube-api-access-rvt6z for pod openshift-network-diagnostics/network-check-target-n658c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:38.814927 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:38.814851    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z podName:6b125d04-b3b2-42a1-8d4e-99c6e5242240 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:46.814830232 +0000 UTC m=+18.169694864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvt6z" (UniqueName: "kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z") pod "network-check-target-n658c" (UID: "6b125d04-b3b2-42a1-8d4e-99c6e5242240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:39.147795 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:39.147685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:39.147947 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:39.147842    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:39.148403 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:39.148376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:39.148517 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:39.148499    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:40.147755 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:40.147718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:40.148243 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:40.147846    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:41.147838 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:41.147754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:41.147838 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:41.147786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:41.148349 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:41.147894    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:41.148349 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:41.147970    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:42.147727 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:42.147698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:42.147862 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:42.147792    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:43.147644 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:43.147611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:43.147802 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:43.147716    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:43.147876 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:43.147804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:43.148157 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:43.147906    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:44.147620 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:44.147591    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:44.147782 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:44.147686    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:45.147926 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:45.147895    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:45.148455 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:45.148014    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:45.148528 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:45.147901    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:45.150810 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:45.148766    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:46.147476 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:46.147444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:46.147679 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:46.147562    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:46.770074 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:46.770039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:46.770497 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:46.770096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:46.770497 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:46.770178    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:46.770497 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:46.770201    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:46.770497 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:46.770235    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret podName:012ca71e-d6e0-45af-a910-83d05cc0dd6b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:02.77021916 +0000 UTC m=+34.125083776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret") pod "global-pull-secret-syncer-96kgg" (UID: "012ca71e-d6e0-45af-a910-83d05cc0dd6b") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:46.770497 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:46.770249    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs podName:80becee4-0e8e-437c-955b-e9cb44506e22 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:02.770242598 +0000 UTC m=+34.125107214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs") pod "network-metrics-daemon-dwzxx" (UID: "80becee4-0e8e-437c-955b-e9cb44506e22") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:46.870759 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:46.870698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:46.870884 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:46.870863    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:46.870941 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:46.870889    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:46.870941 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:46.870900    2561 projected.go:196] Error preparing data for projected volume kube-api-access-rvt6z for pod openshift-network-diagnostics/network-check-target-n658c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:46.871021 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:46.870949    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z podName:6b125d04-b3b2-42a1-8d4e-99c6e5242240 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:02.870931351 +0000 UTC m=+34.225795968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvt6z" (UniqueName: "kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z") pod "network-check-target-n658c" (UID: "6b125d04-b3b2-42a1-8d4e-99c6e5242240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:47.147084 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:47.147008    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:47.147240 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:47.147148    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:47.147240 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:47.147214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:47.147367 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:47.147341    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:48.147663 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:48.147630    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:48.148077 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:48.147736    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:49.148940 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:49.148883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:49.149417 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:49.148985    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:49.149417 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:49.149081    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:49.149417 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:49.149168    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:50.148020 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:50.147986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:50.148199 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:50.148109    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:51.148103 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.147944    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:51.148825 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.148011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:51.148825 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:51.148204    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:51.148825 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:51.148240    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:51.340563 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.340487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" event={"ID":"3dd8fb7f-cee2-4625-af27-51acec6ecdc6","Type":"ContainerStarted","Data":"e644ca56b2ff42f599888c487cbf25ce2324c90069ea7c09c8fad7e847fb096a"}
Sep 16 18:40:51.340757 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.340573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" event={"ID":"3dd8fb7f-cee2-4625-af27-51acec6ecdc6","Type":"ContainerStarted","Data":"2c0400a79fad9966ea974b4a5518c59394cdb6cd97d708d9539059acb61afe92"}
Sep 16 18:40:51.340757 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.340624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" event={"ID":"3dd8fb7f-cee2-4625-af27-51acec6ecdc6","Type":"ContainerStarted","Data":"1e57d23a2979acdbe00445e740ac8d70c8b3cb9908f004adb067347cc527cc95"}
Sep 16 18:40:51.340757 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.340661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" event={"ID":"3dd8fb7f-cee2-4625-af27-51acec6ecdc6","Type":"ContainerStarted","Data":"6461edb82099807be291fb0a55edda6e98036b4c6b9705b87926db53cd364bf5"}
Sep 16 18:40:51.340757 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.340717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" event={"ID":"3dd8fb7f-cee2-4625-af27-51acec6ecdc6","Type":"ContainerStarted","Data":"1f7517310b42b5cb97387fb2b4383cb389f162f8237c51a23b852d163828cb1b"}
Sep 16 18:40:51.340971 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.340758    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" event={"ID":"3dd8fb7f-cee2-4625-af27-51acec6ecdc6","Type":"ContainerStarted","Data":"75e4fb92663b221bec85d50992ae0b51a4424ae147586842c57b61627bc94e10"}
Sep 16 18:40:51.342287 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.342202    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhjgx" event={"ID":"0d4e3ca8-8af4-492f-8d27-68d663b861fe","Type":"ContainerStarted","Data":"7a5e3c83ac4c5ee501fceb704d3d30631648d101b26c03419d73b621c75c7ab0"}
Sep 16 18:40:51.344002 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.343968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gf5lr" event={"ID":"43c13f0d-db2b-46e7-87b6-50fa4666087a","Type":"ContainerStarted","Data":"72a4f6d7d6a245d9a5432055b34d3058e85490e7060b9b357e77f23e47e0808e"}
Sep 16 18:40:51.345741 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.345720    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vxjwf" event={"ID":"b84db093-de67-44ec-ade5-7a4a84648e0d","Type":"ContainerStarted","Data":"4a4062548ef2e2de35eb90a21029f927b8850138a02f62bb0366c7eed9acb93c"}
Sep 16 18:40:51.347416 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.347394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb" event={"ID":"d40797dc-8666-4bf3-9eca-3436c9271385","Type":"ContainerStarted","Data":"3fb4175af76cb41df8ec63e09501ef06d26704bb8e74a016bd20011fdb35c22b"}
Sep 16 18:40:51.349187 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.349163    2561 generic.go:356] "Generic (PLEG): container finished" podID="e2a384ac-d60b-4bbd-872d-dc42760a3455" containerID="a1d564d5a5e02c7a18fe2c90b7978df35f7e192a2f3823eca714f44c0ce04e45" exitCode=0
Sep 16 18:40:51.349289 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.349248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pp8w7" event={"ID":"e2a384ac-d60b-4bbd-872d-dc42760a3455","Type":"ContainerDied","Data":"a1d564d5a5e02c7a18fe2c90b7978df35f7e192a2f3823eca714f44c0ce04e45"}
Sep 16 18:40:51.351030 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.350932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vslg4" event={"ID":"13da0f1f-abdc-4b10-a33d-25aff53b9c84","Type":"ContainerStarted","Data":"10151013ae283fcbac3bdd7cae34996d110d4f6afe964ada444aaaa6b2c29585"}
Sep 16 18:40:51.352606 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.352503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l8qxv" event={"ID":"73d882ea-dbac-48fe-9406-ad43a64e7170","Type":"ContainerStarted","Data":"5e8bc82a2028d5adf9b2923149c3b9a5be6fd92bc89d3a7d68787191a5c7f4e3"}
Sep 16 18:40:51.377638 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.377602    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" podStartSLOduration=20.377591154 podStartE2EDuration="20.377591154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:34.329575632 +0000 UTC m=+5.684440275" watchObservedRunningTime="2026-09-16 18:40:51.377591154 +0000 UTC m=+22.732455793"
Sep 16 18:40:51.402876 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.402841    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xhjgx" podStartSLOduration=3.113054381 podStartE2EDuration="21.402830464s" totalImagesPullingTime="18.289776083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.378149275 +0000 UTC m=+22.733013912" watchObservedRunningTime="2026-09-16 18:40:51.402830464 +0000 UTC m=+22.757695101"
Sep 16 18:40:51.402957 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.402927    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vxjwf" podStartSLOduration=3.109585435 podStartE2EDuration="21.402921175s" totalImagesPullingTime="18.29333574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.401948269 +0000 UTC m=+22.756812907" watchObservedRunningTime="2026-09-16 18:40:51.402921175 +0000 UTC m=+22.757785812"
Sep 16 18:40:51.430969 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.430931    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l8qxv" podStartSLOduration=3.130840355 podStartE2EDuration="21.430921593s" totalImagesPullingTime="18.300081238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.430870285 +0000 UTC m=+22.785734924" watchObservedRunningTime="2026-09-16 18:40:51.430921593 +0000 UTC m=+22.785786230"
Sep 16 18:40:51.457194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.457161    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gf5lr" podStartSLOduration=3.112960028 podStartE2EDuration="21.457151222s" totalImagesPullingTime="18.344191194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.456313408 +0000 UTC m=+22.811178038" watchObservedRunningTime="2026-09-16 18:40:51.457151222 +0000 UTC m=+22.812015860"
Sep 16 18:40:51.481264 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.481229    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vslg4" podStartSLOduration=7.096353016 podStartE2EDuration="21.481211683s" totalImagesPullingTime="14.384858667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.480513261 +0000 UTC m=+22.835377900" watchObservedRunningTime="2026-09-16 18:40:51.481211683 +0000 UTC m=+22.836076322"
Sep 16 18:40:51.816065 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.816034    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.832816 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.832800    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:51.995483 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:51.995459    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:40:52.029377 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.029344    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:52.147679 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.147620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:52.147817 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:52.147755    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:52.148133 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.148063    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:40:51.995478406Z","UUID":"4cedf122-3663-4198-a814-6b2c316516ed","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:40:52.149643 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.149625    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:40:52.149737 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.149658    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:40:52.360104 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.360070    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vcd4n" event={"ID":"54562fc0-f30a-4a41-bef2-dd2b3d51924b","Type":"ContainerStarted","Data":"11fc98f1407156d15a42f8ea4d5c65e7fec0b11ca76ea5a97c1cd4a932431906"}
Sep 16 18:40:52.362417 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.362353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb" event={"ID":"d40797dc-8666-4bf3-9eca-3436c9271385","Type":"ContainerStarted","Data":"cfb57cede249cf9fce9869f1914a7183f7f7d1f5dccb461653db3665850d70b6"}
Sep 16 18:40:52.397254 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.397208    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vcd4n" podStartSLOduration=4.109234479 podStartE2EDuration="22.397192116s" totalImagesPullingTime="18.287957637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:52.396980894 +0000 UTC m=+23.751845532" watchObservedRunningTime="2026-09-16 18:40:52.397192116 +0000 UTC m=+23.752056756"
Sep 16 18:40:52.638544 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.638481    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:52.918186 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:52.918106    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:53.147604 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:53.147577    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:53.147722 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:53.147620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:53.147722 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:53.147693    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:53.148057 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:53.148033    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:53.368780 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:53.368743    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" event={"ID":"3dd8fb7f-cee2-4625-af27-51acec6ecdc6","Type":"ContainerStarted","Data":"55a2ef47d63181cf172228bf802172dfe32890a98ef1db7ab8fb553cf82f6b91"}
Sep 16 18:40:53.371173 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:53.371117    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb" event={"ID":"d40797dc-8666-4bf3-9eca-3436c9271385","Type":"ContainerStarted","Data":"15e32e61e15813a32ceec383a5f7382d6fed498460937229f8a28fb43b271f37"}
Sep 16 18:40:53.404100 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:53.404052    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gvtdb" podStartSLOduration=2.80971359 podStartE2EDuration="23.404032115s" totalImagesPullingTime="20.594318525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:53.400322919 +0000 UTC m=+24.755187556" watchObservedRunningTime="2026-09-16 18:40:53.404032115 +0000 UTC m=+24.758896755"
Sep 16 18:40:53.574796 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:53.574761    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:53.865356 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:53.865271    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:53.987863 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:53.987829    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:54.147683 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:54.147609    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:54.147808 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:54.147722    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:54.356807 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:54.356769    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:54.516167 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:54.516137    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:55.148057 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:55.148028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:55.148219 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:55.148139    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:55.148308 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:55.148217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:55.148371 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:55.148346    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:56.147797 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.147637    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:56.148241 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:56.147865    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:56.289789 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.289765    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:56.290457 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.290437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:56.382174 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.382143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" event={"ID":"3dd8fb7f-cee2-4625-af27-51acec6ecdc6","Type":"ContainerStarted","Data":"2dfdcc4a3dc17bcf6e91b9d7ab4aa836bcecc934dd6369fddf12e59a48acf63c"}
Sep 16 18:40:56.382383 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.382339    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:56.382383 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.382383    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:56.384479 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.384451    2561 generic.go:356] "Generic (PLEG): container finished" podID="e2a384ac-d60b-4bbd-872d-dc42760a3455" containerID="2b8f230e9ffd1968b3f6491393b09650d041cc5eb8b8bc714dec523dcf243550" exitCode=0
Sep 16 18:40:56.384591 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.384524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pp8w7" event={"ID":"e2a384ac-d60b-4bbd-872d-dc42760a3455","Type":"ContainerDied","Data":"2b8f230e9ffd1968b3f6491393b09650d041cc5eb8b8bc714dec523dcf243550"}
Sep 16 18:40:56.385583 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.385139    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vslg4"
Sep 16 18:40:56.398287 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.398237    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:56.428860 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.428819    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt" podStartSLOduration=8.081145497 podStartE2EDuration="26.428805412s" totalImagesPullingTime="18.347659915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:56.428594907 +0000 UTC m=+27.783459545" watchObservedRunningTime="2026-09-16 18:40:56.428805412 +0000 UTC m=+27.783670412"
Sep 16 18:40:56.652753 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:56.652681    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:57.147651 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.147619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:57.147795 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.147626    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:57.147795 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:57.147759    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:57.148313 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:57.147807    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:57.387184 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.387158    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:57.394914 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.394896    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n658c"]
Sep 16 18:40:57.394999 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.394989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:57.395070 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:57.395054    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:57.397431 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.397409    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-96kgg"]
Sep 16 18:40:57.397524 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.397500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:57.397643 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:57.397625    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:57.400250 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.400196    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dwzxx"]
Sep 16 18:40:57.400368 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.400313    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:57.400431 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:57.400414    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:57.403080 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:57.403047    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:40:58.391278 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:58.391242    2561 generic.go:356] "Generic (PLEG): container finished" podID="e2a384ac-d60b-4bbd-872d-dc42760a3455" containerID="0e352957f9228dbdfb72910cc7847c17880d7cbde1980076c2aca36d77359cce" exitCode=0
Sep 16 18:40:58.391719 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:58.391335    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pp8w7" event={"ID":"e2a384ac-d60b-4bbd-872d-dc42760a3455","Type":"ContainerDied","Data":"0e352957f9228dbdfb72910cc7847c17880d7cbde1980076c2aca36d77359cce"}
Sep 16 18:40:59.148381 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:59.148359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:40:59.148475 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:59.148439    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:40:59.148553 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:59.148533    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:40:59.148680 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:59.148662    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:40:59.148747 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:59.148733    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:40:59.148787 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:40:59.148657    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:40:59.395793 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:59.395767    2561 generic.go:356] "Generic (PLEG): container finished" podID="e2a384ac-d60b-4bbd-872d-dc42760a3455" containerID="503f80a09897373b767153df7b4e5be4bd188276ab77531b3430156280b15e41" exitCode=0
Sep 16 18:40:59.396083 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:40:59.395851    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pp8w7" event={"ID":"e2a384ac-d60b-4bbd-872d-dc42760a3455","Type":"ContainerDied","Data":"503f80a09897373b767153df7b4e5be4bd188276ab77531b3430156280b15e41"}
Sep 16 18:41:00.105081 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:00.105042    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:00.251410 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:00.251374    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:01.147866 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:01.147834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:41:01.148305 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:01.147833    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:41:01.148305 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:01.147957    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:41:01.148305 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:01.147843    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:41:01.148305 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:01.148076    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:41:01.148305 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:01.148160    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:41:02.797181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:02.797149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:41:02.797690 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:02.797197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:41:02.797690 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:02.797309    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:02.797690 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:02.797321    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:02.797690 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:02.797356    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret podName:012ca71e-d6e0-45af-a910-83d05cc0dd6b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.79734393 +0000 UTC m=+66.152208562 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret") pod "global-pull-secret-syncer-96kgg" (UID: "012ca71e-d6e0-45af-a910-83d05cc0dd6b") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:02.797690 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:02.797370    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs podName:80becee4-0e8e-437c-955b-e9cb44506e22 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.797363493 +0000 UTC m=+66.152228109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs") pod "network-metrics-daemon-dwzxx" (UID: "80becee4-0e8e-437c-955b-e9cb44506e22") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:02.897925 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:02.897888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:41:02.898102 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:02.898033    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:41:02.898102 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:02.898061    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:41:02.898102 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:02.898088    2561 projected.go:196] Error preparing data for projected volume kube-api-access-rvt6z for pod openshift-network-diagnostics/network-check-target-n658c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:02.898262 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:02.898137    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z podName:6b125d04-b3b2-42a1-8d4e-99c6e5242240 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.898120389 +0000 UTC m=+66.252985026 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rvt6z" (UniqueName: "kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z") pod "network-check-target-n658c" (UID: "6b125d04-b3b2-42a1-8d4e-99c6e5242240") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:03.150367 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.150230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:41:03.150504 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.150374    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:41:03.150504 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.150389    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:41:03.150504 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:03.150383    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-96kgg" podUID="012ca71e-d6e0-45af-a910-83d05cc0dd6b"
Sep 16 18:41:03.150504 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:03.150488    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n658c" podUID="6b125d04-b3b2-42a1-8d4e-99c6e5242240"
Sep 16 18:41:03.150721 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:03.150587    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dwzxx" podUID="80becee4-0e8e-437c-955b-e9cb44506e22"
Sep 16 18:41:03.436237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.436188    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeReady"
Sep 16 18:41:03.436362 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.436285    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:41:03.483076 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.483055    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cbcfd5659-f22bh"]
Sep 16 18:41:03.509008 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.508986    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cbcfd5659-f22bh"]
Sep 16 18:41:03.509119 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.509015    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mhrgc"]
Sep 16 18:41:03.509119 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.509101    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.512097 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.512078    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6l2zg\""
Sep 16 18:41:03.512214 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.512079    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:41:03.512214 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.512158    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:41:03.512214 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.512168    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:41:03.518448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.518310    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:41:03.524956 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.524941    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nnnm2"]
Sep 16 18:41:03.525068 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.525053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhrgc"
Sep 16 18:41:03.528194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.528172    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:41:03.528305 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.528230    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:41:03.528305 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.528271    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xc5gc\""
Sep 16 18:41:03.528305 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.528272    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:41:03.547335 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.547318    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nnnm2"]
Sep 16 18:41:03.547421 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.547347    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhrgc"]
Sep 16 18:41:03.547421 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.547361    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.550163 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.550145    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:41:03.550417 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.550405    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9jkmg\""
Sep 16 18:41:03.550577 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.550552    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:41:03.603237 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/271b52f7-d4bc-4b96-8fc7-5b32c834c886-ca-trust-extracted\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.603365 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-bound-sa-token\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.603365 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-image-registry-private-configuration\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.603365 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603325    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/469beb06-0eab-4e87-8a9d-23da328d232e-cert\") pod \"ingress-canary-mhrgc\" (UID: \"469beb06-0eab-4e87-8a9d-23da328d232e\") " pod="openshift-ingress-canary/ingress-canary-mhrgc"
Sep 16 18:41:03.603532 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-installation-pull-secrets\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.603532 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjs4\" (UniqueName: \"kubernetes.io/projected/469beb06-0eab-4e87-8a9d-23da328d232e-kube-api-access-5pjs4\") pod \"ingress-canary-mhrgc\" (UID: \"469beb06-0eab-4e87-8a9d-23da328d232e\") " pod="openshift-ingress-canary/ingress-canary-mhrgc"
Sep 16 18:41:03.603532 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbjd\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-kube-api-access-gbbjd\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.603532 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-certificates\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.603532 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.603532 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.603518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-trusted-ca\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704280 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btfbf\" (UniqueName: \"kubernetes.io/projected/2fca5262-86ac-490a-b880-b2f620c6b019-kube-api-access-btfbf\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.704377 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-certificates\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704431 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704465 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-trusted-ca\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704509 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/271b52f7-d4bc-4b96-8fc7-5b32c834c886-ca-trust-extracted\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704542 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-bound-sa-token\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704587 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:03.704554    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:03.704587 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:03.704567    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbcfd5659-f22bh: secret "image-registry-tls" not found
Sep 16 18:41:03.704696 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:03.704611    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls podName:271b52f7-d4bc-4b96-8fc7-5b32c834c886 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:04.204598661 +0000 UTC m=+35.559463281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls") pod "image-registry-6cbcfd5659-f22bh" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886") : secret "image-registry-tls" not found
Sep 16 18:41:03.704696 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-image-registry-private-configuration\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704696 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/469beb06-0eab-4e87-8a9d-23da328d232e-cert\") pod \"ingress-canary-mhrgc\" (UID: \"469beb06-0eab-4e87-8a9d-23da328d232e\") " pod="openshift-ingress-canary/ingress-canary-mhrgc"
Sep 16 18:41:03.704856 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-installation-pull-secrets\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704856 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-certificates\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704856 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjs4\" (UniqueName: \"kubernetes.io/projected/469beb06-0eab-4e87-8a9d-23da328d232e-kube-api-access-5pjs4\") pod \"ingress-canary-mhrgc\" (UID: \"469beb06-0eab-4e87-8a9d-23da328d232e\") " pod="openshift-ingress-canary/ingress-canary-mhrgc"
Sep 16 18:41:03.704856 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.704856 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbjd\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-kube-api-access-gbbjd\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.704856 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fca5262-86ac-490a-b880-b2f620c6b019-tmp-dir\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.705138 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fca5262-86ac-490a-b880-b2f620c6b019-config-volume\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.705138 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.704877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/271b52f7-d4bc-4b96-8fc7-5b32c834c886-ca-trust-extracted\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.705404 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.705385    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-trusted-ca\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.708320 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.708281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-installation-pull-secrets\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.708320 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.708318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/469beb06-0eab-4e87-8a9d-23da328d232e-cert\") pod \"ingress-canary-mhrgc\" (UID: \"469beb06-0eab-4e87-8a9d-23da328d232e\") " pod="openshift-ingress-canary/ingress-canary-mhrgc"
Sep 16 18:41:03.708438 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.708283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-image-registry-private-configuration\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.719425 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.719400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-bound-sa-token\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.720410 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.720390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjs4\" (UniqueName: \"kubernetes.io/projected/469beb06-0eab-4e87-8a9d-23da328d232e-kube-api-access-5pjs4\") pod \"ingress-canary-mhrgc\" (UID: \"469beb06-0eab-4e87-8a9d-23da328d232e\") " pod="openshift-ingress-canary/ingress-canary-mhrgc"
Sep 16 18:41:03.720668 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.720651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbjd\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-kube-api-access-gbbjd\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:03.801777 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.801746    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:03.805328 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.805286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.805391 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.805357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fca5262-86ac-490a-b880-b2f620c6b019-tmp-dir\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.805428 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:03.805392    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:03.805465 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:03.805455    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls podName:2fca5262-86ac-490a-b880-b2f620c6b019 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:04.305439511 +0000 UTC m=+35.660304131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls") pod "dns-default-nnnm2" (UID: "2fca5262-86ac-490a-b880-b2f620c6b019") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:03.805516 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.805390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fca5262-86ac-490a-b880-b2f620c6b019-config-volume\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.805571 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.805556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfbf\" (UniqueName: \"kubernetes.io/projected/2fca5262-86ac-490a-b880-b2f620c6b019-kube-api-access-btfbf\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.805683 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.805663    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fca5262-86ac-490a-b880-b2f620c6b019-tmp-dir\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.806039 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.806019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fca5262-86ac-490a-b880-b2f620c6b019-config-volume\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.819008 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.818988    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfbf\" (UniqueName: \"kubernetes.io/projected/2fca5262-86ac-490a-b880-b2f620c6b019-kube-api-access-btfbf\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:03.833063 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:03.833039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhrgc"
Sep 16 18:41:04.110612 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:04.110583    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:04.154849 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:04.154824    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhrgc"]
Sep 16 18:41:04.161015 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:04.160990    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469beb06_0eab_4e87_8a9d_23da328d232e.slice/crio-a68a81ea1ac9769fa9c049613bb6574f951078b565319ab50df101b75baafa8d WatchSource:0}: Error finding container a68a81ea1ac9769fa9c049613bb6574f951078b565319ab50df101b75baafa8d: Status 404 returned error can't find the container with id a68a81ea1ac9769fa9c049613bb6574f951078b565319ab50df101b75baafa8d
Sep 16 18:41:04.207180 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:04.207156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:04.207343 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:04.207328    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:04.207391 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:04.207344    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbcfd5659-f22bh: secret "image-registry-tls" not found
Sep 16 18:41:04.207424 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:04.207393    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls podName:271b52f7-d4bc-4b96-8fc7-5b32c834c886 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.207376879 +0000 UTC m=+36.562241505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls") pod "image-registry-6cbcfd5659-f22bh" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886") : secret "image-registry-tls" not found
Sep 16 18:41:04.308262 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:04.308239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:04.308373 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:04.308360    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:04.308427 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:04.308416    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls podName:2fca5262-86ac-490a-b880-b2f620c6b019 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.308402142 +0000 UTC m=+36.663266758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls") pod "dns-default-nnnm2" (UID: "2fca5262-86ac-490a-b880-b2f620c6b019") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:04.411903 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:04.411849    2561 generic.go:356] "Generic (PLEG): container finished" podID="e2a384ac-d60b-4bbd-872d-dc42760a3455" containerID="abef47e6a3687abfdcd2d1f4c32eec0d2bfad1e6d190c5fb3975ab9e970069ac" exitCode=0
Sep 16 18:41:04.411992 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:04.411910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pp8w7" event={"ID":"e2a384ac-d60b-4bbd-872d-dc42760a3455","Type":"ContainerDied","Data":"abef47e6a3687abfdcd2d1f4c32eec0d2bfad1e6d190c5fb3975ab9e970069ac"}
Sep 16 18:41:04.413180 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:04.413152    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhrgc" event={"ID":"469beb06-0eab-4e87-8a9d-23da328d232e","Type":"ContainerStarted","Data":"a68a81ea1ac9769fa9c049613bb6574f951078b565319ab50df101b75baafa8d"}
Sep 16 18:41:05.147638 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.147606    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:41:05.148374 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.147607    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:41:05.148374 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.147619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:41:05.153724 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.153708    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:05.154842 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.154819    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:05.154842 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.154831    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ndh2n\""
Sep 16 18:41:05.155006 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.154827    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:05.155162 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.155145    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5qbcq\""
Sep 16 18:41:05.155241 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.155186    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:05.216429 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.216405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:05.216589 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:05.216571    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:05.216589 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:05.216593    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbcfd5659-f22bh: secret "image-registry-tls" not found
Sep 16 18:41:05.216698 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:05.216661    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls podName:271b52f7-d4bc-4b96-8fc7-5b32c834c886 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:07.216643233 +0000 UTC m=+38.571507854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls") pod "image-registry-6cbcfd5659-f22bh" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886") : secret "image-registry-tls" not found
Sep 16 18:41:05.317499 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.317475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:05.317620 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:05.317591    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.317675 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:05.317648    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls podName:2fca5262-86ac-490a-b880-b2f620c6b019 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:07.317631239 +0000 UTC m=+38.672495861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls") pod "dns-default-nnnm2" (UID: "2fca5262-86ac-490a-b880-b2f620c6b019") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.418916 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.418860    2561 generic.go:356] "Generic (PLEG): container finished" podID="e2a384ac-d60b-4bbd-872d-dc42760a3455" containerID="be5f135fde113ba096d1140766abc1add6dbbb23d7cb57c7fb3b9d889bf46a89" exitCode=0
Sep 16 18:41:05.419012 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:05.418915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pp8w7" event={"ID":"e2a384ac-d60b-4bbd-872d-dc42760a3455","Type":"ContainerDied","Data":"be5f135fde113ba096d1140766abc1add6dbbb23d7cb57c7fb3b9d889bf46a89"}
Sep 16 18:41:06.424925 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:06.424895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pp8w7" event={"ID":"e2a384ac-d60b-4bbd-872d-dc42760a3455","Type":"ContainerStarted","Data":"4ca6affad4419b22e2b3a36e4cdd18caf2a5fc3598d46ac87180f1121a3b748d"}
Sep 16 18:41:06.462539 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:06.462500    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pp8w7" podStartSLOduration=8.097814411 podStartE2EDuration="36.462486762s" totalImagesPullingTime="28.364672351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:06.460962319 +0000 UTC m=+37.815826958" watchObservedRunningTime="2026-09-16 18:41:06.462486762 +0000 UTC m=+37.817351399"
Sep 16 18:41:06.847336 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:06.847281    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:07.134985 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:07.134957    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:07.230097 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:07.230067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:07.230233 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:07.230209    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:07.230233 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:07.230226    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbcfd5659-f22bh: secret "image-registry-tls" not found
Sep 16 18:41:07.230358 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:07.230286    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls podName:271b52f7-d4bc-4b96-8fc7-5b32c834c886 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:11.230268968 +0000 UTC m=+42.585133588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls") pod "image-registry-6cbcfd5659-f22bh" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886") : secret "image-registry-tls" not found
Sep 16 18:41:07.331034 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:07.331008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:07.331171 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:07.331155    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:07.331214 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:07.331207    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls podName:2fca5262-86ac-490a-b880-b2f620c6b019 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:11.331195969 +0000 UTC m=+42.686060584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls") pod "dns-default-nnnm2" (UID: "2fca5262-86ac-490a-b880-b2f620c6b019") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:07.357382 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:07.357352    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:07.428619 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:07.428560    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhrgc" event={"ID":"469beb06-0eab-4e87-8a9d-23da328d232e","Type":"ContainerStarted","Data":"5845c6a934aaad915f19f93a583fddf2198fd632732c8814367b22894448a186"}
Sep 16 18:41:07.652946 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:07.652914    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:07.884013 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:07.883978    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:08.445278 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.445182    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mhrgc" podStartSLOduration=2.971416278 podStartE2EDuration="5.445162234s" totalImagesPullingTime="2.473745956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:07.459128695 +0000 UTC m=+38.813993333" watchObservedRunningTime="2026-09-16 18:41:08.445162234 +0000 UTC m=+39.800026874"
Sep 16 18:41:08.446051 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.446024    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pxv54"]
Sep 16 18:41:08.471872 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.471840    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.474273 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.474249    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pxv54"]
Sep 16 18:41:08.480486 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.480465    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:41:08.480974 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.480958    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:41:08.481992 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.481964    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:41:08.482086 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.481989    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d8hzw\""
Sep 16 18:41:08.482086 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.482031    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:41:08.638690 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.638669    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5"]
Sep 16 18:41:08.639190 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.639161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-crio-socket\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.639286 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.639204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7fn7\" (UniqueName: \"kubernetes.io/projected/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-api-access-c7fn7\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.639286 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.639244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.639286 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.639284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-data-volume\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.639421 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.639313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.652177 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.652155    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5"
Sep 16 18:41:08.655348 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.655316    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5"]
Sep 16 18:41:08.655703 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.655682    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:41:08.655799 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.655702    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:08.656148 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.656130    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fjl5f\""
Sep 16 18:41:08.739971 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.739913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.739971 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.739964    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-data-volume\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.740152 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.739991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.740152 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.740041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-crio-socket\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.740152 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:08.740054    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:08.740152 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.740061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7bp\" (UniqueName: \"kubernetes.io/projected/69e62a94-7d95-4502-a51e-79c01704a516-kube-api-access-xz7bp\") pod \"migrator-8b4d697c5-8skc5\" (UID: \"69e62a94-7d95-4502-a51e-79c01704a516\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5"
Sep 16 18:41:08.740152 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:08.740121    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:09.240102285 +0000 UTC m=+40.594966920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:08.740152 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:08.740123    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:08.740441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.740175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7fn7\" (UniqueName: \"kubernetes.io/projected/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-api-access-c7fn7\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.740441 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:08.740205    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:09.240188788 +0000 UTC m=+40.595053403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:08.740441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.740213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-data-volume\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.740441 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.740226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-crio-socket\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.755235 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.755205    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7fn7\" (UniqueName: \"kubernetes.io/projected/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-api-access-c7fn7\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:08.841025 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.841000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7bp\" (UniqueName: \"kubernetes.io/projected/69e62a94-7d95-4502-a51e-79c01704a516-kube-api-access-xz7bp\") pod \"migrator-8b4d697c5-8skc5\" (UID: \"69e62a94-7d95-4502-a51e-79c01704a516\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5"
Sep 16 18:41:08.853359 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.853338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7bp\" (UniqueName: \"kubernetes.io/projected/69e62a94-7d95-4502-a51e-79c01704a516-kube-api-access-xz7bp\") pod \"migrator-8b4d697c5-8skc5\" (UID: \"69e62a94-7d95-4502-a51e-79c01704a516\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5"
Sep 16 18:41:08.961313 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:08.961274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5"
Sep 16 18:41:09.243768 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:09.243739    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:09.243900 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:09.243779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:09.243900 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:09.243855    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:09.243900 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:09.243878    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:09.244003 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:09.243914    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:10.243898327 +0000 UTC m=+41.598762963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:09.244003 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:09.243930    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:10.243923599 +0000 UTC m=+41.598788215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:09.287764 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:09.287739    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5"]
Sep 16 18:41:09.290521 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:09.290498    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e62a94_7d95_4502_a51e_79c01704a516.slice/crio-a6b1f455211561723a3ca0e7425dbad6ae99e555588d13e2d961ec6a85a0c889 WatchSource:0}: Error finding container a6b1f455211561723a3ca0e7425dbad6ae99e555588d13e2d961ec6a85a0c889: Status 404 returned error can't find the container with id a6b1f455211561723a3ca0e7425dbad6ae99e555588d13e2d961ec6a85a0c889
Sep 16 18:41:09.433787 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:09.433757    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5" event={"ID":"69e62a94-7d95-4502-a51e-79c01704a516","Type":"ContainerStarted","Data":"a6b1f455211561723a3ca0e7425dbad6ae99e555588d13e2d961ec6a85a0c889"}
Sep 16 18:41:09.610037 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:09.609985    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l8qxv_73d882ea-dbac-48fe-9406-ad43a64e7170/dns-node-resolver/0.log"
Sep 16 18:41:10.252596 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.252563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:10.252743 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.252602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:10.252743 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:10.252703    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:10.252743 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:10.252724    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:10.252855 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:10.252762    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:12.252745972 +0000 UTC m=+43.607610609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:10.252855 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:10.252776    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:12.252770007 +0000 UTC m=+43.607634624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:10.637119 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.637090    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-prvzx"]
Sep 16 18:41:10.649856 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.649837    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.653090 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.653065    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:41:10.653184 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.653067    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:41:10.653184 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.653166    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:41:10.654530 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.654515    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:41:10.654596 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.654569    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vdbfl\""
Sep 16 18:41:10.719415 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.719391    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-prvzx"]
Sep 16 18:41:10.756964 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.756936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6-signing-cabundle\") pod \"service-ca-568869fdc5-prvzx\" (UID: \"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6\") " pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.757037 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.756965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9xv\" (UniqueName: \"kubernetes.io/projected/f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6-kube-api-access-cj9xv\") pod \"service-ca-568869fdc5-prvzx\" (UID: \"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6\") " pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.757037 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.756993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6-signing-key\") pod \"service-ca-568869fdc5-prvzx\" (UID: \"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6\") " pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.858251 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.858228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6-signing-cabundle\") pod \"service-ca-568869fdc5-prvzx\" (UID: \"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6\") " pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.858351 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.858253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9xv\" (UniqueName: \"kubernetes.io/projected/f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6-kube-api-access-cj9xv\") pod \"service-ca-568869fdc5-prvzx\" (UID: \"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6\") " pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.858351 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.858282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6-signing-key\") pod \"service-ca-568869fdc5-prvzx\" (UID: \"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6\") " pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.858961 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.858942    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6-signing-cabundle\") pod \"service-ca-568869fdc5-prvzx\" (UID: \"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6\") " pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.860724 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.860704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6-signing-key\") pod \"service-ca-568869fdc5-prvzx\" (UID: \"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6\") " pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.868538 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.868514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9xv\" (UniqueName: \"kubernetes.io/projected/f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6-kube-api-access-cj9xv\") pod \"service-ca-568869fdc5-prvzx\" (UID: \"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6\") " pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:10.958984 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:10.958943    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-prvzx"
Sep 16 18:41:11.017635 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:11.017618    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vxjwf_b84db093-de67-44ec-ade5-7a4a84648e0d/node-ca/0.log"
Sep 16 18:41:11.261859 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:11.261833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:11.261972 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:11.261953    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:11.262009 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:11.261977    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbcfd5659-f22bh: secret "image-registry-tls" not found
Sep 16 18:41:11.262042 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:11.262028    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls podName:271b52f7-d4bc-4b96-8fc7-5b32c834c886 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.262014582 +0000 UTC m=+50.616879211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls") pod "image-registry-6cbcfd5659-f22bh" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886") : secret "image-registry-tls" not found
Sep 16 18:41:11.275175 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:11.275153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-prvzx"]
Sep 16 18:41:11.277373 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:11.277350    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fde55c_1259_49ce_bb4e_b0fcddcfcdb6.slice/crio-8865ad1a7a5e05a1f3f14950ae544872d2d7021c2fe668304336ea5102677a83 WatchSource:0}: Error finding container 8865ad1a7a5e05a1f3f14950ae544872d2d7021c2fe668304336ea5102677a83: Status 404 returned error can't find the container with id 8865ad1a7a5e05a1f3f14950ae544872d2d7021c2fe668304336ea5102677a83
Sep 16 18:41:11.362469 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:11.362446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:11.362590 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:11.362574    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:11.362633 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:11.362625    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls podName:2fca5262-86ac-490a-b880-b2f620c6b019 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:19.362611794 +0000 UTC m=+50.717476410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls") pod "dns-default-nnnm2" (UID: "2fca5262-86ac-490a-b880-b2f620c6b019") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:11.412733 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:11.412712    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mhrgc_469beb06-0eab-4e87-8a9d-23da328d232e/serve-healthcheck-canary/0.log"
Sep 16 18:41:11.439561 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:11.439531    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-prvzx" event={"ID":"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6","Type":"ContainerStarted","Data":"8865ad1a7a5e05a1f3f14950ae544872d2d7021c2fe668304336ea5102677a83"}
Sep 16 18:41:11.441285 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:11.441267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5" event={"ID":"69e62a94-7d95-4502-a51e-79c01704a516","Type":"ContainerStarted","Data":"6fe7b3c98e6d8833371cacd4afc7a8f6c92e24ede9cb2c136a59a40d68d8a481"}
Sep 16 18:41:11.441391 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:11.441333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5" event={"ID":"69e62a94-7d95-4502-a51e-79c01704a516","Type":"ContainerStarted","Data":"1ee0dfa8397415524ae8f76411aba52d51e44d3e864012454a21fc789a363475"}
Sep 16 18:41:11.478968 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:11.478904    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-8skc5" podStartSLOduration=1.76403474 podStartE2EDuration="3.47889105s" totalImagesPullingTime="1.71485631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:11.477933094 +0000 UTC m=+42.832797732" watchObservedRunningTime="2026-09-16 18:41:11.47889105 +0000 UTC m=+42.833755685"
Sep 16 18:41:12.269744 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:12.269707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:12.270168 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:12.269767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:12.270168 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:12.269886    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:12.270168 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:12.269969    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:16.269949397 +0000 UTC m=+47.624814031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:12.270168 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:12.269885    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:12.270168 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:12.270050    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:16.270032985 +0000 UTC m=+47.624897620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:14.450390 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:14.450355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-prvzx" event={"ID":"f0fde55c-1259-49ce-bb4e-b0fcddcfcdb6","Type":"ContainerStarted","Data":"cdd0d5cf87de3eced4066d06495586e78d5141e6fe0d92a565cefee7fb1d629e"}
Sep 16 18:41:14.503843 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:14.503798    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-prvzx" podStartSLOduration=2.366652077 podStartE2EDuration="4.503784448s" totalImagesPullingTime="2.137132371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:14.503542593 +0000 UTC m=+45.858407231" watchObservedRunningTime="2026-09-16 18:41:14.503784448 +0000 UTC m=+45.858649086"
Sep 16 18:41:16.299813 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:16.299776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:16.300242 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:16.299826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:16.300242 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:16.299923    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:16.300242 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:16.299934    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:16.300242 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:16.299997    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:24.299978424 +0000 UTC m=+55.654843057 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:16.300242 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:16.300013    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:24.300007446 +0000 UTC m=+55.654872066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:19.322324 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:19.322276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:19.324677 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:19.324651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") pod \"image-registry-6cbcfd5659-f22bh\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") " pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:19.419955 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:19.419931    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:19.423129 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:19.423109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:19.425112 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:19.425087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fca5262-86ac-490a-b880-b2f620c6b019-metrics-tls\") pod \"dns-default-nnnm2\" (UID: \"2fca5262-86ac-490a-b880-b2f620c6b019\") " pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:19.455451 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:19.455431    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:19.763882 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:19.763862    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cbcfd5659-f22bh"]
Sep 16 18:41:19.766957 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:19.766933    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271b52f7_d4bc_4b96_8fc7_5b32c834c886.slice/crio-4f23268e9bd779ba885589634e8787f682a3ba5ba5bbae8bc500143bb5a143e1 WatchSource:0}: Error finding container 4f23268e9bd779ba885589634e8787f682a3ba5ba5bbae8bc500143bb5a143e1: Status 404 returned error can't find the container with id 4f23268e9bd779ba885589634e8787f682a3ba5ba5bbae8bc500143bb5a143e1
Sep 16 18:41:19.772492 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:19.772468    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nnnm2"]
Sep 16 18:41:19.774948 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:19.774924    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fca5262_86ac_490a_b880_b2f620c6b019.slice/crio-62297e0b484642e61e60cbe40877026a7785ade9ec8236cd7a73733bacce8d08 WatchSource:0}: Error finding container 62297e0b484642e61e60cbe40877026a7785ade9ec8236cd7a73733bacce8d08: Status 404 returned error can't find the container with id 62297e0b484642e61e60cbe40877026a7785ade9ec8236cd7a73733bacce8d08
Sep 16 18:41:20.466464 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:20.466428    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nnnm2" event={"ID":"2fca5262-86ac-490a-b880-b2f620c6b019","Type":"ContainerStarted","Data":"62297e0b484642e61e60cbe40877026a7785ade9ec8236cd7a73733bacce8d08"}
Sep 16 18:41:20.468371 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:20.468339    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh" event={"ID":"271b52f7-d4bc-4b96-8fc7-5b32c834c886","Type":"ContainerStarted","Data":"1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb"}
Sep 16 18:41:20.468524 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:20.468387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh" event={"ID":"271b52f7-d4bc-4b96-8fc7-5b32c834c886","Type":"ContainerStarted","Data":"4f23268e9bd779ba885589634e8787f682a3ba5ba5bbae8bc500143bb5a143e1"}
Sep 16 18:41:20.468524 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:20.468440    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:20.514955 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:20.514910    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh" podStartSLOduration=21.514894965 podStartE2EDuration="21.514894965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:20.513325014 +0000 UTC m=+51.868189651" watchObservedRunningTime="2026-09-16 18:41:20.514894965 +0000 UTC m=+51.869759602"
Sep 16 18:41:22.476763 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:22.476735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nnnm2" event={"ID":"2fca5262-86ac-490a-b880-b2f620c6b019","Type":"ContainerStarted","Data":"9a0fbd3bfb1e9c7acf92f5ecd82f89ce426426c0c3d8178ab6dfc464bd3f2623"}
Sep 16 18:41:22.477090 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:22.476801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nnnm2" event={"ID":"2fca5262-86ac-490a-b880-b2f620c6b019","Type":"ContainerStarted","Data":"85d821e313240496b6198e52fcfef7efc19b7f66e4c0b1b97a2da4cb46943510"}
Sep 16 18:41:22.477090 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:22.476867    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:22.521896 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:22.521852    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nnnm2" podStartSLOduration=17.852788277 podStartE2EDuration="19.521839459s" totalImagesPullingTime="1.669051182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:22.521682718 +0000 UTC m=+53.876547356" watchObservedRunningTime="2026-09-16 18:41:22.521839459 +0000 UTC m=+53.876704096"
Sep 16 18:41:24.360242 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:24.360205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:24.360677 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:24.360260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:24.360677 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:24.360330    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:24.360677 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:24.360383    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls podName:817be5e4-45cb-4cb7-9aff-5f7b674aedc4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:40.360369518 +0000 UTC m=+71.715234134 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxv54" (UID: "817be5e4-45cb-4cb7-9aff-5f7b674aedc4") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:24.360801 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:24.360737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:29.406086 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:29.406054    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z7clt"
Sep 16 18:41:32.482029 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:32.481997    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nnnm2"
Sep 16 18:41:33.323059 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.323025    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"]
Sep 16 18:41:33.360785 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.360753    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"]
Sep 16 18:41:33.360956 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.360860    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.363992 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.363959    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:41:33.364520 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.364496    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:41:33.364757 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.364717    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:41:33.365501 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.365483    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:41:33.372678 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.372649    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"]
Sep 16 18:41:33.372813 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.372793    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"
Sep 16 18:41:33.380279 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.380242    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:41:33.380415 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.380371    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4q5mc\""
Sep 16 18:41:33.388820 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.388794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"]
Sep 16 18:41:33.388935 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.388834    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"]
Sep 16 18:41:33.388935 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.388846    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"]
Sep 16 18:41:33.389006 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.388956    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.392726 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.392705    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:41:33.392726 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.392719    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:41:33.393053 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.393036    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:41:33.393129 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.393066    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:41:33.406185 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.406154    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cbcfd5659-f22bh"]
Sep 16 18:41:33.422024 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.421991    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkrc\" (UniqueName: \"kubernetes.io/projected/e3b876c3-db08-4481-ba7b-287834dee3cf-kube-api-access-cvkrc\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.422152 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80a65bbe-7587-4e39-ad67-637ad23ce296-tmp\") pod \"klusterlet-addon-workmgr-b4f594f4f-slfz6\" (UID: \"80a65bbe-7587-4e39-ad67-637ad23ce296\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.422152 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b876c3-db08-4481-ba7b-287834dee3cf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.422152 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-ca\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.422267 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.422267 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.422267 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6240b1fb-0281-4f32-a087-06ff8350f34d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-656b4759c4-6kprx\" (UID: \"6240b1fb-0281-4f32-a087-06ff8350f34d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"
Sep 16 18:41:33.422385 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/80a65bbe-7587-4e39-ad67-637ad23ce296-klusterlet-config\") pod \"klusterlet-addon-workmgr-b4f594f4f-slfz6\" (UID: \"80a65bbe-7587-4e39-ad67-637ad23ce296\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.422385 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422342    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fkxz\" (UniqueName: \"kubernetes.io/projected/6240b1fb-0281-4f32-a087-06ff8350f34d-kube-api-access-4fkxz\") pod \"managed-serviceaccount-addon-agent-656b4759c4-6kprx\" (UID: \"6240b1fb-0281-4f32-a087-06ff8350f34d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"
Sep 16 18:41:33.422385 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp6dx\" (UniqueName: \"kubernetes.io/projected/80a65bbe-7587-4e39-ad67-637ad23ce296-kube-api-access-lp6dx\") pod \"klusterlet-addon-workmgr-b4f594f4f-slfz6\" (UID: \"80a65bbe-7587-4e39-ad67-637ad23ce296\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.422476 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.422394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-hub\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.523569 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/80a65bbe-7587-4e39-ad67-637ad23ce296-klusterlet-config\") pod \"klusterlet-addon-workmgr-b4f594f4f-slfz6\" (UID: \"80a65bbe-7587-4e39-ad67-637ad23ce296\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fkxz\" (UniqueName: \"kubernetes.io/projected/6240b1fb-0281-4f32-a087-06ff8350f34d-kube-api-access-4fkxz\") pod \"managed-serviceaccount-addon-agent-656b4759c4-6kprx\" (UID: \"6240b1fb-0281-4f32-a087-06ff8350f34d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6dx\" (UniqueName: \"kubernetes.io/projected/80a65bbe-7587-4e39-ad67-637ad23ce296-kube-api-access-lp6dx\") pod \"klusterlet-addon-workmgr-b4f594f4f-slfz6\" (UID: \"80a65bbe-7587-4e39-ad67-637ad23ce296\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-hub\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkrc\" (UniqueName: \"kubernetes.io/projected/e3b876c3-db08-4481-ba7b-287834dee3cf-kube-api-access-cvkrc\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80a65bbe-7587-4e39-ad67-637ad23ce296-tmp\") pod \"klusterlet-addon-workmgr-b4f594f4f-slfz6\" (UID: \"80a65bbe-7587-4e39-ad67-637ad23ce296\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b876c3-db08-4481-ba7b-287834dee3cf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-ca\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.523966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.523903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6240b1fb-0281-4f32-a087-06ff8350f34d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-656b4759c4-6kprx\" (UID: \"6240b1fb-0281-4f32-a087-06ff8350f34d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"
Sep 16 18:41:33.524447 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.524172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80a65bbe-7587-4e39-ad67-637ad23ce296-tmp\") pod \"klusterlet-addon-workmgr-b4f594f4f-slfz6\" (UID: \"80a65bbe-7587-4e39-ad67-637ad23ce296\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.524598 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.524575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b876c3-db08-4481-ba7b-287834dee3cf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.526763 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.526733    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-ca\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.526890 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.526794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6240b1fb-0281-4f32-a087-06ff8350f34d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-656b4759c4-6kprx\" (UID: \"6240b1fb-0281-4f32-a087-06ff8350f34d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"
Sep 16 18:41:33.526890 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.526807    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/80a65bbe-7587-4e39-ad67-637ad23ce296-klusterlet-config\") pod \"klusterlet-addon-workmgr-b4f594f4f-slfz6\" (UID: \"80a65bbe-7587-4e39-ad67-637ad23ce296\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.526890 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.526826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.526890 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.526867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-hub\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.527085 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.527069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e3b876c3-db08-4481-ba7b-287834dee3cf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.542289 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.542266    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86ddc94b5d-ktgfr"]
Sep 16 18:41:33.574588 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.574523    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.577271 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.577245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6dx\" (UniqueName: \"kubernetes.io/projected/80a65bbe-7587-4e39-ad67-637ad23ce296-kube-api-access-lp6dx\") pod \"klusterlet-addon-workmgr-b4f594f4f-slfz6\" (UID: \"80a65bbe-7587-4e39-ad67-637ad23ce296\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.577395 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.577333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkrc\" (UniqueName: \"kubernetes.io/projected/e3b876c3-db08-4481-ba7b-287834dee3cf-kube-api-access-cvkrc\") pod \"cluster-proxy-proxy-agent-79547d6d49-sdxwl\" (UID: \"e3b876c3-db08-4481-ba7b-287834dee3cf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.578482 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.578465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fkxz\" (UniqueName: \"kubernetes.io/projected/6240b1fb-0281-4f32-a087-06ff8350f34d-kube-api-access-4fkxz\") pod \"managed-serviceaccount-addon-agent-656b4759c4-6kprx\" (UID: \"6240b1fb-0281-4f32-a087-06ff8350f34d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"
Sep 16 18:41:33.624454 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.624421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxhs\" (UniqueName: \"kubernetes.io/projected/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-kube-api-access-7rxhs\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.624581 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.624465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-installation-pull-secrets\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.624581 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.624482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-bound-sa-token\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.624654 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.624584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-image-registry-private-configuration\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.624654 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.624627    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-registry-certificates\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.624730 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.624711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-ca-trust-extracted\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.624774 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.624761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-registry-tls\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.624807 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.624782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-trusted-ca\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.640718 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.640682    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86ddc94b5d-ktgfr"]
Sep 16 18:41:33.671088 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.671054    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:33.694385 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.694361    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"
Sep 16 18:41:33.702538 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.702503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"
Sep 16 18:41:33.726089 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.726055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-trusted-ca\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.726226 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.726122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxhs\" (UniqueName: \"kubernetes.io/projected/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-kube-api-access-7rxhs\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.726226 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.726150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-installation-pull-secrets\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.726226 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.726169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-bound-sa-token\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.726226 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.726197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-image-registry-private-configuration\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.726226 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.726223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-registry-certificates\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.726513 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.726268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-ca-trust-extracted\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.726513 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.726314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-registry-tls\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.727033 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.726974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-ca-trust-extracted\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.727210 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.727182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-trusted-ca\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.727287 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.727214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-registry-certificates\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.729398 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.729370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-installation-pull-secrets\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.729510 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.729410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-registry-tls\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.729836 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.729809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-image-registry-private-configuration\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.765732 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.765666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-bound-sa-token\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.771524 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.771392    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxhs\" (UniqueName: \"kubernetes.io/projected/49cf8e26-f602-4ae6-84c3-e979a00ee6a9-kube-api-access-7rxhs\") pod \"image-registry-86ddc94b5d-ktgfr\" (UID: \"49cf8e26-f602-4ae6-84c3-e979a00ee6a9\") " pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:33.906610 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:33.906515    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:34.010647 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.010618    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"]
Sep 16 18:41:34.017378 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:34.017353    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a65bbe_7587_4e39_ad67_637ad23ce296.slice/crio-7e58c96352e15696bb1b8ac17e78a2e01f412438d32431f28edf88af507623e1 WatchSource:0}: Error finding container 7e58c96352e15696bb1b8ac17e78a2e01f412438d32431f28edf88af507623e1: Status 404 returned error can't find the container with id 7e58c96352e15696bb1b8ac17e78a2e01f412438d32431f28edf88af507623e1
Sep 16 18:41:34.041923 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.041902    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx"]
Sep 16 18:41:34.043675 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:34.043653    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6240b1fb_0281_4f32_a087_06ff8350f34d.slice/crio-93d6659511ec1bb9bf4c6f73688583289c08b39afe8e530c6fc58e520e713727 WatchSource:0}: Error finding container 93d6659511ec1bb9bf4c6f73688583289c08b39afe8e530c6fc58e520e713727: Status 404 returned error can't find the container with id 93d6659511ec1bb9bf4c6f73688583289c08b39afe8e530c6fc58e520e713727
Sep 16 18:41:34.053509 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.053491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl"]
Sep 16 18:41:34.055080 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:34.055061    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3b876c3_db08_4481_ba7b_287834dee3cf.slice/crio-09218e40d39cc909f86004d771a9456f0ab0e1fb2e798a87c8bc6fde410c2375 WatchSource:0}: Error finding container 09218e40d39cc909f86004d771a9456f0ab0e1fb2e798a87c8bc6fde410c2375: Status 404 returned error can't find the container with id 09218e40d39cc909f86004d771a9456f0ab0e1fb2e798a87c8bc6fde410c2375
Sep 16 18:41:34.297959 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.297923    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86ddc94b5d-ktgfr"]
Sep 16 18:41:34.310218 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:34.310187    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cf8e26_f602_4ae6_84c3_e979a00ee6a9.slice/crio-747ab1d85c7491a6993adc44e232a79bb4d00e5dc59190c030be408b79dbdc51 WatchSource:0}: Error finding container 747ab1d85c7491a6993adc44e232a79bb4d00e5dc59190c030be408b79dbdc51: Status 404 returned error can't find the container with id 747ab1d85c7491a6993adc44e232a79bb4d00e5dc59190c030be408b79dbdc51
Sep 16 18:41:34.513264 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.513227    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx" event={"ID":"6240b1fb-0281-4f32-a087-06ff8350f34d","Type":"ContainerStarted","Data":"93d6659511ec1bb9bf4c6f73688583289c08b39afe8e530c6fc58e520e713727"}
Sep 16 18:41:34.514425 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.514396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6" event={"ID":"80a65bbe-7587-4e39-ad67-637ad23ce296","Type":"ContainerStarted","Data":"7e58c96352e15696bb1b8ac17e78a2e01f412438d32431f28edf88af507623e1"}
Sep 16 18:41:34.515951 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.515900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr" event={"ID":"49cf8e26-f602-4ae6-84c3-e979a00ee6a9","Type":"ContainerStarted","Data":"2182c04c16db0d1506a74381b6bec603c609f1ba5c9ced5ea7b46697d3dc71cd"}
Sep 16 18:41:34.516070 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.515973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr" event={"ID":"49cf8e26-f602-4ae6-84c3-e979a00ee6a9","Type":"ContainerStarted","Data":"747ab1d85c7491a6993adc44e232a79bb4d00e5dc59190c030be408b79dbdc51"}
Sep 16 18:41:34.516070 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.516001    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:34.517046 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.517027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl" event={"ID":"e3b876c3-db08-4481-ba7b-287834dee3cf","Type":"ContainerStarted","Data":"09218e40d39cc909f86004d771a9456f0ab0e1fb2e798a87c8bc6fde410c2375"}
Sep 16 18:41:34.567359 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.567229    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr" podStartSLOduration=1.567211099 podStartE2EDuration="1.567211099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:34.567189948 +0000 UTC m=+65.922054585" watchObservedRunningTime="2026-09-16 18:41:34.567211099 +0000 UTC m=+65.922075738"
Sep 16 18:41:34.836788 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.836405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:41:34.836788 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.836496    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:41:34.841652 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.841621    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:34.841904 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.841830    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:34.849536 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.849501    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/012ca71e-d6e0-45af-a910-83d05cc0dd6b-original-pull-secret\") pod \"global-pull-secret-syncer-96kgg\" (UID: \"012ca71e-d6e0-45af-a910-83d05cc0dd6b\") " pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:41:34.849976 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.849860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80becee4-0e8e-437c-955b-e9cb44506e22-metrics-certs\") pod \"network-metrics-daemon-dwzxx\" (UID: \"80becee4-0e8e-437c-955b-e9cb44506e22\") " pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:41:34.877659 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.871876    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-96kgg"
Sep 16 18:41:34.877659 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.874591    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ndh2n\""
Sep 16 18:41:34.877659 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.876235    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dwzxx"
Sep 16 18:41:34.939435 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.939390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:41:34.942787 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.942761    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:34.953959 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.953706    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:34.967971 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:34.967914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt6z\" (UniqueName: \"kubernetes.io/projected/6b125d04-b3b2-42a1-8d4e-99c6e5242240-kube-api-access-rvt6z\") pod \"network-check-target-n658c\" (UID: \"6b125d04-b3b2-42a1-8d4e-99c6e5242240\") " pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:41:35.165500 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:35.165423    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5qbcq\""
Sep 16 18:41:35.170449 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:35.170422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:41:35.202025 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:35.201724    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-96kgg"]
Sep 16 18:41:35.234932 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:35.233433    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dwzxx"]
Sep 16 18:41:35.241685 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:35.241649    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80becee4_0e8e_437c_955b_e9cb44506e22.slice/crio-1ab9133179ada64cc98e434e111c6b620127f9142b2bcff255810730f1b8c278 WatchSource:0}: Error finding container 1ab9133179ada64cc98e434e111c6b620127f9142b2bcff255810730f1b8c278: Status 404 returned error can't find the container with id 1ab9133179ada64cc98e434e111c6b620127f9142b2bcff255810730f1b8c278
Sep 16 18:41:35.499835 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:35.499664    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n658c"]
Sep 16 18:41:35.504413 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:35.504373    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b125d04_b3b2_42a1_8d4e_99c6e5242240.slice/crio-fd4041e90ad56944f3abcb29682bf4f234d76fe3dc6119cbdffeefb4db06e491 WatchSource:0}: Error finding container fd4041e90ad56944f3abcb29682bf4f234d76fe3dc6119cbdffeefb4db06e491: Status 404 returned error can't find the container with id fd4041e90ad56944f3abcb29682bf4f234d76fe3dc6119cbdffeefb4db06e491
Sep 16 18:41:35.539804 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:35.539701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n658c" event={"ID":"6b125d04-b3b2-42a1-8d4e-99c6e5242240","Type":"ContainerStarted","Data":"fd4041e90ad56944f3abcb29682bf4f234d76fe3dc6119cbdffeefb4db06e491"}
Sep 16 18:41:35.542112 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:35.542079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-96kgg" event={"ID":"012ca71e-d6e0-45af-a910-83d05cc0dd6b","Type":"ContainerStarted","Data":"62ce30b0189c7a39b4199173d19710ac43e5e0058c9c74824f7a7af9c488a8fb"}
Sep 16 18:41:35.548084 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:35.548054    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dwzxx" event={"ID":"80becee4-0e8e-437c-955b-e9cb44506e22","Type":"ContainerStarted","Data":"1ab9133179ada64cc98e434e111c6b620127f9142b2bcff255810730f1b8c278"}
Sep 16 18:41:40.397800 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:40.397763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:40.400278 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:40.400251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/817be5e4-45cb-4cb7-9aff-5f7b674aedc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxv54\" (UID: \"817be5e4-45cb-4cb7-9aff-5f7b674aedc4\") " pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:40.584964 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:40.584931    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d8hzw\""
Sep 16 18:41:40.592869 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:40.592839    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pxv54"
Sep 16 18:41:43.412789 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:43.412746    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:44.517479 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.517393    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pxv54"]
Sep 16 18:41:44.521400 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:44.521350    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817be5e4_45cb_4cb7_9aff_5f7b674aedc4.slice/crio-65c5c9ea9269cc45e6da1a054d24277a8778051fdb0c3725e2cb8095925dda4b WatchSource:0}: Error finding container 65c5c9ea9269cc45e6da1a054d24277a8778051fdb0c3725e2cb8095925dda4b: Status 404 returned error can't find the container with id 65c5c9ea9269cc45e6da1a054d24277a8778051fdb0c3725e2cb8095925dda4b
Sep 16 18:41:44.591018 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.590921    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl" event={"ID":"e3b876c3-db08-4481-ba7b-287834dee3cf","Type":"ContainerStarted","Data":"50e5e14da15deb522a3a3c52ace6190fbaa9869bea51763012074c2de654d953"}
Sep 16 18:41:44.591018 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.590973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl" event={"ID":"e3b876c3-db08-4481-ba7b-287834dee3cf","Type":"ContainerStarted","Data":"2010a651b22153777a04c82355cf2158192adc5665c3cbf429b856640393b406"}
Sep 16 18:41:44.591018 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.590992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl" event={"ID":"e3b876c3-db08-4481-ba7b-287834dee3cf","Type":"ContainerStarted","Data":"0915a9842c2a7261a41c2e6c8cbb113d7d91ab50941c576d2e383a1ebff56d58"}
Sep 16 18:41:44.593777 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.593753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx" event={"ID":"6240b1fb-0281-4f32-a087-06ff8350f34d","Type":"ContainerStarted","Data":"4e62b91593825eacabe4c9ed13bc93d5bd1e331efca4726a182b8b26cc095503"}
Sep 16 18:41:44.595842 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.595726    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6" event={"ID":"80a65bbe-7587-4e39-ad67-637ad23ce296","Type":"ContainerStarted","Data":"11ab5325c1eb1090914ead9e1d5c59b892e1699f79b09887db8e8f5c5431e74d"}
Sep 16 18:41:44.596016 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.595973    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:44.597980 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.597954    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n658c" event={"ID":"6b125d04-b3b2-42a1-8d4e-99c6e5242240","Type":"ContainerStarted","Data":"2e15344771dd218b082f2e80df93d34ef153176192df1ed221f8b67e0847cebe"}
Sep 16 18:41:44.598265 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.598211    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6"
Sep 16 18:41:44.598387 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.598369    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:41:44.603546 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.603501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-96kgg" event={"ID":"012ca71e-d6e0-45af-a910-83d05cc0dd6b","Type":"ContainerStarted","Data":"762085760d5de39f90723c0bc68ddf7f9919970905f17c58b1470e3b905cc91f"}
Sep 16 18:41:44.605443 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.605338    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxv54" event={"ID":"817be5e4-45cb-4cb7-9aff-5f7b674aedc4","Type":"ContainerStarted","Data":"3bcf11b5b43d73ab8e4b09348f648571e62c55dae1bb37139d54ac685cde1f09"}
Sep 16 18:41:44.605443 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.605389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxv54" event={"ID":"817be5e4-45cb-4cb7-9aff-5f7b674aedc4","Type":"ContainerStarted","Data":"65c5c9ea9269cc45e6da1a054d24277a8778051fdb0c3725e2cb8095925dda4b"}
Sep 16 18:41:44.607601 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.607577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dwzxx" event={"ID":"80becee4-0e8e-437c-955b-e9cb44506e22","Type":"ContainerStarted","Data":"58f2a5b5b8ac3ee69110f0d4b65a788b47bbd99bd62ec1f94c14591dee1326f9"}
Sep 16 18:41:44.607686 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.607621    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dwzxx" event={"ID":"80becee4-0e8e-437c-955b-e9cb44506e22","Type":"ContainerStarted","Data":"78bfe00183ac8256c61755f434eb52b43f084e954e6adfa127a2dea0d12f53f6"}
Sep 16 18:41:44.628111 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.628061    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79547d6d49-sdxwl" podStartSLOduration=1.433445534 podStartE2EDuration="11.628044004s" totalImagesPullingTime="10.19459847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:44.625589517 +0000 UTC m=+75.980454155" watchObservedRunningTime="2026-09-16 18:41:44.628044004 +0000 UTC m=+75.982908644"
Sep 16 18:41:44.708346 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.708248    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dwzxx" podStartSLOduration=65.73367031 podStartE2EDuration="1m14.708231817s" totalImagesPullingTime="8.974561507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:44.667828773 +0000 UTC m=+76.022693411" watchObservedRunningTime="2026-09-16 18:41:44.708231817 +0000 UTC m=+76.063096455"
Sep 16 18:41:44.708725 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.708687    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-96kgg" podStartSLOduration=65.637939601 podStartE2EDuration="1m14.70867425s" totalImagesPullingTime="9.070734649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:44.708312162 +0000 UTC m=+76.063176799" watchObservedRunningTime="2026-09-16 18:41:44.70867425 +0000 UTC m=+76.063538889"
Sep 16 18:41:44.750116 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.750039    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n658c" podStartSLOduration=66.00604238 podStartE2EDuration="1m14.750024139s" totalImagesPullingTime="8.743981759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:44.748965892 +0000 UTC m=+76.103830535" watchObservedRunningTime="2026-09-16 18:41:44.750024139 +0000 UTC m=+76.104888777"
Sep 16 18:41:44.784904 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.784870    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b4f594f4f-slfz6" podStartSLOduration=1.55259638 podStartE2EDuration="11.784857312s" totalImagesPullingTime="10.232260932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:44.784669087 +0000 UTC m=+76.139533725" watchObservedRunningTime="2026-09-16 18:41:44.784857312 +0000 UTC m=+76.139721940"
Sep 16 18:41:44.826039 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:44.825995    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-656b4759c4-6kprx" podStartSLOduration=1.650399266 podStartE2EDuration="11.825982585s" totalImagesPullingTime="10.175583319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:44.824833061 +0000 UTC m=+76.179697707" watchObservedRunningTime="2026-09-16 18:41:44.825982585 +0000 UTC m=+76.180847223"
Sep 16 18:41:45.154387 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.154361    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n6l4w"]
Sep 16 18:41:45.157938 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.157918    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.161769 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.161550    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nr5ld\""
Sep 16 18:41:45.162448 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.162425    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:41:45.162657 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.162639    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:41:45.162720 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.162656    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:41:45.162904 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.162884    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:41:45.163738 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.163719    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:41:45.165562 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.165395    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:41:45.239980 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.239952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-textfile\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.240128 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.239992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/456725a2-50b4-402d-b137-062def6de1f8-metrics-client-ca\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.240128 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.240028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-wtmp\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.240128 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.240089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/456725a2-50b4-402d-b137-062def6de1f8-sys\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.240308 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.240125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.240308 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.240179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-tls\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.240308 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.240206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhwb\" (UniqueName: \"kubernetes.io/projected/456725a2-50b4-402d-b137-062def6de1f8-kube-api-access-7dhwb\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.240308 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.240231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.240308 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.240259    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/456725a2-50b4-402d-b137-062def6de1f8-root\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.341591 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.341563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/456725a2-50b4-402d-b137-062def6de1f8-root\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.341711 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.341610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-textfile\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.341711 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.341638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/456725a2-50b4-402d-b137-062def6de1f8-metrics-client-ca\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.341711 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.341673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-wtmp\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.341711 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.341706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/456725a2-50b4-402d-b137-062def6de1f8-sys\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.341911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.341729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.341911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.341809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-tls\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.341911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.341838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhwb\" (UniqueName: \"kubernetes.io/projected/456725a2-50b4-402d-b137-062def6de1f8-kube-api-access-7dhwb\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.341911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.341867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.342287 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.342178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/456725a2-50b4-402d-b137-062def6de1f8-root\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.342425 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:45.342404    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:41:45.342486 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.342426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.342486 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:45.342468    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-tls podName:456725a2-50b4-402d-b137-062def6de1f8 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.842450818 +0000 UTC m=+77.197315439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-tls") pod "node-exporter-n6l4w" (UID: "456725a2-50b4-402d-b137-062def6de1f8") : secret "node-exporter-tls" not found
Sep 16 18:41:45.342657 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.342636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/456725a2-50b4-402d-b137-062def6de1f8-sys\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.342791 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.342773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-wtmp\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.343245 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.343194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-textfile\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.344179 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.343486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/456725a2-50b4-402d-b137-062def6de1f8-metrics-client-ca\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.346966 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.346918    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.368928 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.368908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhwb\" (UniqueName: \"kubernetes.io/projected/456725a2-50b4-402d-b137-062def6de1f8-kube-api-access-7dhwb\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.619049 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.619012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxv54" event={"ID":"817be5e4-45cb-4cb7-9aff-5f7b674aedc4","Type":"ContainerStarted","Data":"74043cbdefb7f0d9be2c38e962649498a08cd09ffece48b0e66238bba6e48f9f"}
Sep 16 18:41:45.846871 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.846834    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-tls\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:45.849446 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:45.849417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/456725a2-50b4-402d-b137-062def6de1f8-node-exporter-tls\") pod \"node-exporter-n6l4w\" (UID: \"456725a2-50b4-402d-b137-062def6de1f8\") " pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:46.069471 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:46.069434    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n6l4w"
Sep 16 18:41:46.083228 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:41:46.083196    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456725a2_50b4_402d_b137_062def6de1f8.slice/crio-6f65b7309015af78f9bcc3de74e61cc669e6f1644e0703757c2ae1ac82b13dd7 WatchSource:0}: Error finding container 6f65b7309015af78f9bcc3de74e61cc669e6f1644e0703757c2ae1ac82b13dd7: Status 404 returned error can't find the container with id 6f65b7309015af78f9bcc3de74e61cc669e6f1644e0703757c2ae1ac82b13dd7
Sep 16 18:41:46.623769 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:46.623728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6l4w" event={"ID":"456725a2-50b4-402d-b137-062def6de1f8","Type":"ContainerStarted","Data":"6f65b7309015af78f9bcc3de74e61cc669e6f1644e0703757c2ae1ac82b13dd7"}
Sep 16 18:41:47.629459 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:47.629417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxv54" event={"ID":"817be5e4-45cb-4cb7-9aff-5f7b674aedc4","Type":"ContainerStarted","Data":"edcf78f15d6a725a30f2173a74fdbdfa85e9c7d76bfe79614ca0682a9ca5bbca"}
Sep 16 18:41:47.631082 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:47.631053    2561 generic.go:356] "Generic (PLEG): container finished" podID="456725a2-50b4-402d-b137-062def6de1f8" containerID="1d7c33b526d776515fd7a37b2b283bd5bf29734cc4e3db35561bd2fb14a19ad2" exitCode=0
Sep 16 18:41:47.631193 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:47.631145    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6l4w" event={"ID":"456725a2-50b4-402d-b137-062def6de1f8","Type":"ContainerDied","Data":"1d7c33b526d776515fd7a37b2b283bd5bf29734cc4e3db35561bd2fb14a19ad2"}
Sep 16 18:41:47.675125 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:47.675079    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pxv54" podStartSLOduration=37.665723424 podStartE2EDuration="39.675062384s" totalImagesPullingTime="2.00933896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:47.674410638 +0000 UTC m=+79.029275275" watchObservedRunningTime="2026-09-16 18:41:47.675062384 +0000 UTC m=+79.029927058"
Sep 16 18:41:48.637398 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:48.637354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6l4w" event={"ID":"456725a2-50b4-402d-b137-062def6de1f8","Type":"ContainerStarted","Data":"4c3f4b50b86d73639cd554398d0d901251fe93390ffd7064915329819b9ad9cd"}
Sep 16 18:41:48.638002 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:48.637411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n6l4w" event={"ID":"456725a2-50b4-402d-b137-062def6de1f8","Type":"ContainerStarted","Data":"a8b247f42c8d35d062d64dade284d15d21ad0623a8a66903c3cf4dac606207f7"}
Sep 16 18:41:48.671169 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:48.671117    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n6l4w" podStartSLOduration=2.5093321570000002 podStartE2EDuration="3.671098528s" totalImagesPullingTime="1.101465874s" totalInitContainerRuntime="60.300497ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:48.668927526 +0000 UTC m=+80.023792167" watchObservedRunningTime="2026-09-16 18:41:48.671098528 +0000 UTC m=+80.025963165"
Sep 16 18:41:55.553004 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:55.552976    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86ddc94b5d-ktgfr"
Sep 16 18:41:58.425908 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.425865    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh" podUID="271b52f7-d4bc-4b96-8fc7-5b32c834c886" containerName="registry" containerID="cri-o://1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb" gracePeriod=30
Sep 16 18:41:58.670433 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.670409    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:58.673445 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.673422    2561 generic.go:356] "Generic (PLEG): container finished" podID="271b52f7-d4bc-4b96-8fc7-5b32c834c886" containerID="1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb" exitCode=0
Sep 16 18:41:58.673513 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.673458    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh"
Sep 16 18:41:58.673513 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.673467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh" event={"ID":"271b52f7-d4bc-4b96-8fc7-5b32c834c886","Type":"ContainerDied","Data":"1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb"}
Sep 16 18:41:58.673513 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.673504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cbcfd5659-f22bh" event={"ID":"271b52f7-d4bc-4b96-8fc7-5b32c834c886","Type":"ContainerDied","Data":"4f23268e9bd779ba885589634e8787f682a3ba5ba5bbae8bc500143bb5a143e1"}
Sep 16 18:41:58.673620 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.673524    2561 scope.go:117] "RemoveContainer" containerID="1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb"
Sep 16 18:41:58.681133 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.681116    2561 scope.go:117] "RemoveContainer" containerID="1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb"
Sep 16 18:41:58.681454 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:41:58.681433    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb\": container with ID starting with 1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb not found: ID does not exist" containerID="1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb"
Sep 16 18:41:58.681505 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.681462    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb"} err="failed to get container status \"1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb\": rpc error: code = NotFound desc = could not find container \"1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb\": container with ID starting with 1531feb8bd4fb5f6c1497da5c7196fa5e4bab886ae1b7ce6479fe16cd14f74fb not found: ID does not exist"
Sep 16 18:41:58.752564 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.752546    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-installation-pull-secrets\") pod \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") "
Sep 16 18:41:58.752670 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.752580    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-bound-sa-token\") pod \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") "
Sep 16 18:41:58.752670 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.752599    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-image-registry-private-configuration\") pod \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") "
Sep 16 18:41:58.752670 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.752623    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-trusted-ca\") pod \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") "
Sep 16 18:41:58.752670 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.752645    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-certificates\") pod \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") "
Sep 16 18:41:58.752898 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.752697    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/271b52f7-d4bc-4b96-8fc7-5b32c834c886-ca-trust-extracted\") pod \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") "
Sep 16 18:41:58.752898 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.752725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbbjd\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-kube-api-access-gbbjd\") pod \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") "
Sep 16 18:41:58.752898 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.752759    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") pod \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\" (UID: \"271b52f7-d4bc-4b96-8fc7-5b32c834c886\") "
Sep 16 18:41:58.753134 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.753108    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "271b52f7-d4bc-4b96-8fc7-5b32c834c886" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:58.753134 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.753112    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "271b52f7-d4bc-4b96-8fc7-5b32c834c886" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:58.755146 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.755116    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "271b52f7-d4bc-4b96-8fc7-5b32c834c886" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:41:58.755235 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.755155    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-kube-api-access-gbbjd" (OuterVolumeSpecName: "kube-api-access-gbbjd") pod "271b52f7-d4bc-4b96-8fc7-5b32c834c886" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886"). InnerVolumeSpecName "kube-api-access-gbbjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:58.755290 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.755222    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "271b52f7-d4bc-4b96-8fc7-5b32c834c886" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:41:58.755290 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.755257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "271b52f7-d4bc-4b96-8fc7-5b32c834c886" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:58.755403 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.755387    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "271b52f7-d4bc-4b96-8fc7-5b32c834c886" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:58.761353 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.761327    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271b52f7-d4bc-4b96-8fc7-5b32c834c886-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "271b52f7-d4bc-4b96-8fc7-5b32c834c886" (UID: "271b52f7-d4bc-4b96-8fc7-5b32c834c886"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:58.853805 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.853782    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/271b52f7-d4bc-4b96-8fc7-5b32c834c886-ca-trust-extracted\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:41:58.853805 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.853802    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbjd\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-kube-api-access-gbbjd\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:41:58.853919 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.853812    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:41:58.853919 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.853820    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-installation-pull-secrets\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:41:58.853919 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.853828    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/271b52f7-d4bc-4b96-8fc7-5b32c834c886-bound-sa-token\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:41:58.853919 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.853837    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/271b52f7-d4bc-4b96-8fc7-5b32c834c886-image-registry-private-configuration\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:41:58.853919 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.853846    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-trusted-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:41:58.853919 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.853854    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/271b52f7-d4bc-4b96-8fc7-5b32c834c886-registry-certificates\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:41:58.998833 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:58.998808    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cbcfd5659-f22bh"]
Sep 16 18:41:59.015785 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:59.015761    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cbcfd5659-f22bh"]
Sep 16 18:41:59.152194 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:41:59.152170    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="271b52f7-d4bc-4b96-8fc7-5b32c834c886" path="/var/lib/kubelet/pods/271b52f7-d4bc-4b96-8fc7-5b32c834c886/volumes"
Sep 16 18:42:16.626181 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:42:16.626138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n658c"
Sep 16 18:45:28.944690 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:45:28.944661    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:49:49.778149 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.778113    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"]
Sep 16 18:49:49.778668 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.778474    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271b52f7-d4bc-4b96-8fc7-5b32c834c886" containerName="registry"
Sep 16 18:49:49.778668 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.778493    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="271b52f7-d4bc-4b96-8fc7-5b32c834c886" containerName="registry"
Sep 16 18:49:49.778668 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.778552    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="271b52f7-d4bc-4b96-8fc7-5b32c834c886" containerName="registry"
Sep 16 18:49:49.781461 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.781439    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:49.784504 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.784481    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:49:49.784504 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.784492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-35a06-predictor-serving-cert\""
Sep 16 18:49:49.785790 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.785764    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:49:49.785911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.785831    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-24wmq\""
Sep 16 18:49:49.785911 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.785899    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-35a06-kube-rbac-proxy-sar-config\""
Sep 16 18:49:49.794338 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.794316    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"]
Sep 16 18:49:49.942131 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.942103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9cj7\" (UniqueName: \"kubernetes.io/projected/4eacdac5-055c-4da4-bb90-82838b787b41-kube-api-access-w9cj7\") pod \"error-404-isvc-35a06-predictor-59f949b8b5-wkdgn\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") " pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:49.942259 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.942133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eacdac5-055c-4da4-bb90-82838b787b41-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-35a06-predictor-59f949b8b5-wkdgn\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") " pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:49.942259 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:49.942156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eacdac5-055c-4da4-bb90-82838b787b41-proxy-tls\") pod \"error-404-isvc-35a06-predictor-59f949b8b5-wkdgn\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") " pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:50.042509 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:50.042447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9cj7\" (UniqueName: \"kubernetes.io/projected/4eacdac5-055c-4da4-bb90-82838b787b41-kube-api-access-w9cj7\") pod \"error-404-isvc-35a06-predictor-59f949b8b5-wkdgn\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") " pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:50.042509 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:50.042477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eacdac5-055c-4da4-bb90-82838b787b41-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-35a06-predictor-59f949b8b5-wkdgn\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") " pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:50.042509 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:50.042505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eacdac5-055c-4da4-bb90-82838b787b41-proxy-tls\") pod \"error-404-isvc-35a06-predictor-59f949b8b5-wkdgn\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") " pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:50.043223 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:50.043194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eacdac5-055c-4da4-bb90-82838b787b41-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-35a06-predictor-59f949b8b5-wkdgn\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") " pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:50.045003 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:50.044983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eacdac5-055c-4da4-bb90-82838b787b41-proxy-tls\") pod \"error-404-isvc-35a06-predictor-59f949b8b5-wkdgn\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") " pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:50.054525 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:50.054491    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9cj7\" (UniqueName: \"kubernetes.io/projected/4eacdac5-055c-4da4-bb90-82838b787b41-kube-api-access-w9cj7\") pod \"error-404-isvc-35a06-predictor-59f949b8b5-wkdgn\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") " pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:50.091351 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:50.091331    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:49:50.361601 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:50.361582    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"]
Sep 16 18:49:50.363648 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:49:50.363623    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eacdac5_055c_4da4_bb90_82838b787b41.slice/crio-da4eba3ae8e206aab737fc71730a5eb80103523d9a63829c3cfee4fb29f83686 WatchSource:0}: Error finding container da4eba3ae8e206aab737fc71730a5eb80103523d9a63829c3cfee4fb29f83686: Status 404 returned error can't find the container with id da4eba3ae8e206aab737fc71730a5eb80103523d9a63829c3cfee4fb29f83686
Sep 16 18:49:50.365534 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:50.365516    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:49:51.275783 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:49:51.275745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" event={"ID":"4eacdac5-055c-4da4-bb90-82838b787b41","Type":"ContainerStarted","Data":"da4eba3ae8e206aab737fc71730a5eb80103523d9a63829c3cfee4fb29f83686"}
Sep 16 18:50:02.318420 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:02.318385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" event={"ID":"4eacdac5-055c-4da4-bb90-82838b787b41","Type":"ContainerStarted","Data":"825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43"}
Sep 16 18:50:05.331373 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:05.331340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" event={"ID":"4eacdac5-055c-4da4-bb90-82838b787b41","Type":"ContainerStarted","Data":"bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c"}
Sep 16 18:50:05.331743 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:05.331459    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:50:05.359812 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:05.359774    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podStartSLOduration=2.015037473 podStartE2EDuration="16.359761502s" totalImagesPullingTime="14.344724029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:05.358368004 +0000 UTC m=+576.713232644" watchObservedRunningTime="2026-09-16 18:50:05.359761502 +0000 UTC m=+576.714626140"
Sep 16 18:50:06.335614 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:06.335579    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:50:06.336892 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:06.336854    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 16 18:50:07.339288 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:07.339242    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 16 18:50:12.344420 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:12.344385    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:50:12.345003 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:12.344970    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 16 18:50:22.345797 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:22.345754    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 16 18:50:32.348572 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:32.348517    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 16 18:50:42.345903 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:42.345851    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 16 18:50:52.346133 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:50:52.346100    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:51:19.641625 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.641594    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"]
Sep 16 18:51:19.642136 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.641823    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container" containerID="cri-o://825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43" gracePeriod=30
Sep 16 18:51:19.642136 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.641870    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kube-rbac-proxy" containerID="cri-o://bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c" gracePeriod=30
Sep 16 18:51:19.832256 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.832227    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"]
Sep 16 18:51:19.835656 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.835639    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:19.842409 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.842381    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-15f71-predictor-serving-cert\""
Sep 16 18:51:19.842523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.842487    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-15f71-kube-rbac-proxy-sar-config\""
Sep 16 18:51:19.846130 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.846104    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"]
Sep 16 18:51:19.937354 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.937280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") " pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:19.937354 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.937326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lb5\" (UniqueName: \"kubernetes.io/projected/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-api-access-g4lb5\") pod \"error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") " pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:19.937510 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:19.937357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-proxy-tls\") pod \"error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") " pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.038682 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.038656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lb5\" (UniqueName: \"kubernetes.io/projected/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-api-access-g4lb5\") pod \"error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") " pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.038796 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.038723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-proxy-tls\") pod \"error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") " pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.038796 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.038787    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") " pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.039458 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.039437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") " pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.041009 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.040991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-proxy-tls\") pod \"error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") " pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.047792 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.047768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lb5\" (UniqueName: \"kubernetes.io/projected/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-api-access-g4lb5\") pod \"error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") " pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.147870 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.147845    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.433607 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.423521    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"]
Sep 16 18:51:20.433607 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:51:20.425883    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a04d9ae_1ae6_46ca_a41d_d3f404f8aa6b.slice/crio-1a8b2122423ba8831f0afe9121d516dcf2b6ab327648ddb016085df4bcba96cf WatchSource:0}: Error finding container 1a8b2122423ba8831f0afe9121d516dcf2b6ab327648ddb016085df4bcba96cf: Status 404 returned error can't find the container with id 1a8b2122423ba8831f0afe9121d516dcf2b6ab327648ddb016085df4bcba96cf
Sep 16 18:51:20.581490 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.581458    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" event={"ID":"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b","Type":"ContainerStarted","Data":"1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7"}
Sep 16 18:51:20.581632 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.581517    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.581632 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.581548    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:20.581632 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.581574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" event={"ID":"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b","Type":"ContainerStarted","Data":"e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c"}
Sep 16 18:51:20.581632 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.581600    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" event={"ID":"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b","Type":"ContainerStarted","Data":"1a8b2122423ba8831f0afe9121d516dcf2b6ab327648ddb016085df4bcba96cf"}
Sep 16 18:51:20.583239 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.583206    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 16 18:51:20.583523 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.583501    2561 generic.go:356] "Generic (PLEG): container finished" podID="4eacdac5-055c-4da4-bb90-82838b787b41" containerID="bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c" exitCode=2
Sep 16 18:51:20.583591 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.583551    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" event={"ID":"4eacdac5-055c-4da4-bb90-82838b787b41","Type":"ContainerDied","Data":"bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c"}
Sep 16 18:51:20.602841 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:20.602806    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podStartSLOduration=1.602793227 podStartE2EDuration="1.602793227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:20.601757002 +0000 UTC m=+651.956621639" watchObservedRunningTime="2026-09-16 18:51:20.602793227 +0000 UTC m=+651.957657865"
Sep 16 18:51:21.592119 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:21.592078    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 16 18:51:22.339504 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.339464    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.16:8643/healthz\": dial tcp 10.133.0.16:8643: connect: connection refused"
Sep 16 18:51:22.345996 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.345959    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.16:8080: connect: connection refused"
Sep 16 18:51:22.480517 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.480492    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:51:22.557897 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.557874    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9cj7\" (UniqueName: \"kubernetes.io/projected/4eacdac5-055c-4da4-bb90-82838b787b41-kube-api-access-w9cj7\") pod \"4eacdac5-055c-4da4-bb90-82838b787b41\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") "
Sep 16 18:51:22.557990 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.557930    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eacdac5-055c-4da4-bb90-82838b787b41-proxy-tls\") pod \"4eacdac5-055c-4da4-bb90-82838b787b41\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") "
Sep 16 18:51:22.558045 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.558027    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eacdac5-055c-4da4-bb90-82838b787b41-kube-rbac-proxy-sar-config\") pod \"4eacdac5-055c-4da4-bb90-82838b787b41\" (UID: \"4eacdac5-055c-4da4-bb90-82838b787b41\") "
Sep 16 18:51:22.558442 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.558416    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eacdac5-055c-4da4-bb90-82838b787b41-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4eacdac5-055c-4da4-bb90-82838b787b41" (UID: "4eacdac5-055c-4da4-bb90-82838b787b41"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:22.559951 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.559930    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eacdac5-055c-4da4-bb90-82838b787b41-kube-api-access-w9cj7" (OuterVolumeSpecName: "kube-api-access-w9cj7") pod "4eacdac5-055c-4da4-bb90-82838b787b41" (UID: "4eacdac5-055c-4da4-bb90-82838b787b41"). InnerVolumeSpecName "kube-api-access-w9cj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:22.560033 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.560018    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eacdac5-055c-4da4-bb90-82838b787b41-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4eacdac5-055c-4da4-bb90-82838b787b41" (UID: "4eacdac5-055c-4da4-bb90-82838b787b41"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:22.597066 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.597043    2561 generic.go:356] "Generic (PLEG): container finished" podID="4eacdac5-055c-4da4-bb90-82838b787b41" containerID="825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43" exitCode=0
Sep 16 18:51:22.597356 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.597076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" event={"ID":"4eacdac5-055c-4da4-bb90-82838b787b41","Type":"ContainerDied","Data":"825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43"}
Sep 16 18:51:22.597356 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.597088    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"
Sep 16 18:51:22.597356 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.597149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn" event={"ID":"4eacdac5-055c-4da4-bb90-82838b787b41","Type":"ContainerDied","Data":"da4eba3ae8e206aab737fc71730a5eb80103523d9a63829c3cfee4fb29f83686"}
Sep 16 18:51:22.597356 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.597178    2561 scope.go:117] "RemoveContainer" containerID="bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c"
Sep 16 18:51:22.604986 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.604963    2561 scope.go:117] "RemoveContainer" containerID="825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43"
Sep 16 18:51:22.612491 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.612473    2561 scope.go:117] "RemoveContainer" containerID="bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c"
Sep 16 18:51:22.612776 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:51:22.612755    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c\": container with ID starting with bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c not found: ID does not exist" containerID="bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c"
Sep 16 18:51:22.612854 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.612788    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c"} err="failed to get container status \"bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c\": rpc error: code = NotFound desc = could not find container \"bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c\": container with ID starting with bdfa75faf64ff085b6d3efdcb051d0b1d1ed4d289f4c9985fe05c0aeeb20924c not found: ID does not exist"
Sep 16 18:51:22.612854 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.612810    2561 scope.go:117] "RemoveContainer" containerID="825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43"
Sep 16 18:51:22.613094 ip-10-0-128-29 kubenswrapper[2561]: E0916 18:51:22.613069    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43\": container with ID starting with 825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43 not found: ID does not exist" containerID="825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43"
Sep 16 18:51:22.613146 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.613102    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43"} err="failed to get container status \"825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43\": rpc error: code = NotFound desc = could not find container \"825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43\": container with ID starting with 825c6c6caeeadd8bf5c259f25caca5e6a4fbaa40775feae86001e1ad2f295c43 not found: ID does not exist"
Sep 16 18:51:22.622944 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.622925    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"]
Sep 16 18:51:22.629830 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.629809    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-35a06-predictor-59f949b8b5-wkdgn"]
Sep 16 18:51:22.659249 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.659232    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eacdac5-055c-4da4-bb90-82838b787b41-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:51:22.659249 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.659252    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eacdac5-055c-4da4-bb90-82838b787b41-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:51:22.659384 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:22.659261    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9cj7\" (UniqueName: \"kubernetes.io/projected/4eacdac5-055c-4da4-bb90-82838b787b41-kube-api-access-w9cj7\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 18:51:23.155781 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:23.155749    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" path="/var/lib/kubelet/pods/4eacdac5-055c-4da4-bb90-82838b787b41/volumes"
Sep 16 18:51:26.596054 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:26.596027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:51:26.596586 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:26.596557    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 16 18:51:36.597003 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:36.596959    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 16 18:51:46.596964 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:46.596923    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 16 18:51:56.597057 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:51:56.597015    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 16 18:52:06.597517 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:06.597479    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 18:52:09.792517 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.792486    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"]
Sep 16 18:52:09.792860 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.792752    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kube-rbac-proxy"
Sep 16 18:52:09.792860 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.792762    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kube-rbac-proxy"
Sep 16 18:52:09.792860 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.792770    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container"
Sep 16 18:52:09.792860 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.792775    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container"
Sep 16 18:52:09.792860 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.792818    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kube-rbac-proxy"
Sep 16 18:52:09.792860 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.792825    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eacdac5-055c-4da4-bb90-82838b787b41" containerName="kserve-container"
Sep 16 18:52:09.795702 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.795686    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:09.798560 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.798538    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-21301-predictor-serving-cert\""
Sep 16 18:52:09.798695 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.798584    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-21301-kube-rbac-proxy-sar-config\""
Sep 16 18:52:09.806072 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.806053    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"]
Sep 16 18:52:09.964224 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.964200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21301-predictor-6f496df769-xkcv9\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") " pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:09.964337 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.964231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb524-6850-46d8-85f9-7dce31e6bec7-proxy-tls\") pod \"error-404-isvc-21301-predictor-6f496df769-xkcv9\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") " pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:09.964337 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:09.964250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b429f\" (UniqueName: \"kubernetes.io/projected/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-api-access-b429f\") pod \"error-404-isvc-21301-predictor-6f496df769-xkcv9\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") " pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:10.065012 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.064958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21301-predictor-6f496df769-xkcv9\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") " pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:10.065012 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.064990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb524-6850-46d8-85f9-7dce31e6bec7-proxy-tls\") pod \"error-404-isvc-21301-predictor-6f496df769-xkcv9\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") " pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:10.065138 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.065015    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b429f\" (UniqueName: \"kubernetes.io/projected/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-api-access-b429f\") pod \"error-404-isvc-21301-predictor-6f496df769-xkcv9\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") " pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:10.065697 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.065674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21301-predictor-6f496df769-xkcv9\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") " pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:10.067507 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.067483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb524-6850-46d8-85f9-7dce31e6bec7-proxy-tls\") pod \"error-404-isvc-21301-predictor-6f496df769-xkcv9\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") " pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:10.075099 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.075079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b429f\" (UniqueName: \"kubernetes.io/projected/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-api-access-b429f\") pod \"error-404-isvc-21301-predictor-6f496df769-xkcv9\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") " pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:10.106080 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.106056    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:10.375616 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.375575    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"]
Sep 16 18:52:10.379015 ip-10-0-128-29 kubenswrapper[2561]: W0916 18:52:10.378982    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccddb524_6850_46d8_85f9_7dce31e6bec7.slice/crio-c97a9101f59e72c461d73f24222419389ff424c8ba41141d16f42d28a44c2011 WatchSource:0}: Error finding container c97a9101f59e72c461d73f24222419389ff424c8ba41141d16f42d28a44c2011: Status 404 returned error can't find the container with id c97a9101f59e72c461d73f24222419389ff424c8ba41141d16f42d28a44c2011
Sep 16 18:52:10.759905 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.759828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" event={"ID":"ccddb524-6850-46d8-85f9-7dce31e6bec7","Type":"ContainerStarted","Data":"643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65"}
Sep 16 18:52:10.759905 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.759889    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" event={"ID":"ccddb524-6850-46d8-85f9-7dce31e6bec7","Type":"ContainerStarted","Data":"1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a"}
Sep 16 18:52:10.760063 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.759910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" event={"ID":"ccddb524-6850-46d8-85f9-7dce31e6bec7","Type":"ContainerStarted","Data":"c97a9101f59e72c461d73f24222419389ff424c8ba41141d16f42d28a44c2011"}
Sep 16 18:52:10.760063 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.759943    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:10.780271 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:10.780225    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" podStartSLOduration=1.780211398 podStartE2EDuration="1.780211398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:10.778334671 +0000 UTC m=+702.133199306" watchObservedRunningTime="2026-09-16 18:52:10.780211398 +0000 UTC m=+702.135076036"
Sep 16 18:52:11.763535 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:11.763497    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:11.764511 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:11.764483    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 16 18:52:12.767056 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:12.767016    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 16 18:52:17.771723 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:17.771696    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 18:52:17.772173 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:17.772148    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 16 18:52:27.772481 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:27.772393    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 16 18:52:37.772999 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:37.772953    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 16 18:52:47.772161 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:47.772124    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 16 18:52:57.773476 ip-10-0-128-29 kubenswrapper[2561]: I0916 18:52:57.773431    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 19:00:35.099501 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.099469    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"]
Sep 16 19:00:35.102637 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.102622    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.106847 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.106824    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1593f-predictor-serving-cert\""
Sep 16 19:00:35.107250 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.107234    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1593f-kube-rbac-proxy-sar-config\""
Sep 16 19:00:35.118886 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.118864    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"]
Sep 16 19:00:35.187990 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.187967    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"]
Sep 16 19:00:35.188189 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.188164    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kube-rbac-proxy" containerID="cri-o://1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7" gracePeriod=30
Sep 16 19:00:35.188265 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.188155    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container" containerID="cri-o://e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c" gracePeriod=30
Sep 16 19:00:35.258424 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.258400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-proxy-tls\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.258505 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.258430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.258559 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.258536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z2bv\" (UniqueName: \"kubernetes.io/projected/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-api-access-5z2bv\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.359486 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.359436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z2bv\" (UniqueName: \"kubernetes.io/projected/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-api-access-5z2bv\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.359486 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.359477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-proxy-tls\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.359603 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.359494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.359650 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:00:35.359597    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1593f-predictor-serving-cert: secret "error-404-isvc-1593f-predictor-serving-cert" not found
Sep 16 19:00:35.359699 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:00:35.359654    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-proxy-tls podName:ca4e37bb-749a-42dd-a32a-ef7f2445e1a3 nodeName:}" failed. No retries permitted until 2026-09-16 19:00:35.859637692 +0000 UTC m=+1207.214502311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-proxy-tls") pod "error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" (UID: "ca4e37bb-749a-42dd-a32a-ef7f2445e1a3") : secret "error-404-isvc-1593f-predictor-serving-cert" not found
Sep 16 19:00:35.360022 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.360006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.378115 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.378084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z2bv\" (UniqueName: \"kubernetes.io/projected/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-api-access-5z2bv\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.500319 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.500269    2561 generic.go:356] "Generic (PLEG): container finished" podID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerID="1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7" exitCode=2
Sep 16 19:00:35.500435 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.500331    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" event={"ID":"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b","Type":"ContainerDied","Data":"1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7"}
Sep 16 19:00:35.862506 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.862469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-proxy-tls\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:35.864843 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:35.864825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-proxy-tls\") pod \"error-404-isvc-1593f-predictor-599ddd5c95-jfd8z\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") " pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:36.013991 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.013966    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:36.292436 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.292413    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"]
Sep 16 19:00:36.294068 ip-10-0-128-29 kubenswrapper[2561]: W0916 19:00:36.294041    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4e37bb_749a_42dd_a32a_ef7f2445e1a3.slice/crio-8e0551be842cdfb415b104a9772430d01d6444ccd2057306cc016f96c6c7698f WatchSource:0}: Error finding container 8e0551be842cdfb415b104a9772430d01d6444ccd2057306cc016f96c6c7698f: Status 404 returned error can't find the container with id 8e0551be842cdfb415b104a9772430d01d6444ccd2057306cc016f96c6c7698f
Sep 16 19:00:36.510413 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.510375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" event={"ID":"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3","Type":"ContainerStarted","Data":"9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807"}
Sep 16 19:00:36.510598 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.510456    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" event={"ID":"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3","Type":"ContainerStarted","Data":"338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36"}
Sep 16 19:00:36.510598 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.510478    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" event={"ID":"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3","Type":"ContainerStarted","Data":"8e0551be842cdfb415b104a9772430d01d6444ccd2057306cc016f96c6c7698f"}
Sep 16 19:00:36.510598 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.510517    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:36.510741 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.510646    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:36.512000 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.511973    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 16 19:00:36.536461 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.536422    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podStartSLOduration=1.536411394 podStartE2EDuration="1.536411394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:36.536063857 +0000 UTC m=+1207.890928494" watchObservedRunningTime="2026-09-16 19:00:36.536411394 +0000 UTC m=+1207.891276031"
Sep 16 19:00:36.592833 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.592771    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Sep 16 19:00:36.597446 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:36.597425    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 16 19:00:37.514494 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:37.514453    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 16 19:00:37.927200 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:37.927174    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 19:00:38.076453 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.076393    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-rbac-proxy-sar-config\") pod \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") "
Sep 16 19:00:38.076453 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.076430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4lb5\" (UniqueName: \"kubernetes.io/projected/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-api-access-g4lb5\") pod \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") "
Sep 16 19:00:38.076453 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.076452    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-proxy-tls\") pod \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\" (UID: \"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b\") "
Sep 16 19:00:38.076758 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.076732    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" (UID: "3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:38.078458 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.078435    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-api-access-g4lb5" (OuterVolumeSpecName: "kube-api-access-g4lb5") pod "3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" (UID: "3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b"). InnerVolumeSpecName "kube-api-access-g4lb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:38.078523 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.078494    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" (UID: "3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:38.177014 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.176986    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4lb5\" (UniqueName: \"kubernetes.io/projected/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-api-access-g4lb5\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:00:38.177014 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.177011    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:00:38.177143 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.177028    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:00:38.519680 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.519650    2561 generic.go:356] "Generic (PLEG): container finished" podID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerID="e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c" exitCode=0
Sep 16 19:00:38.520024 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.519684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" event={"ID":"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b","Type":"ContainerDied","Data":"e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c"}
Sep 16 19:00:38.520024 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.519696    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"
Sep 16 19:00:38.520024 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.519729    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8" event={"ID":"3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b","Type":"ContainerDied","Data":"1a8b2122423ba8831f0afe9121d516dcf2b6ab327648ddb016085df4bcba96cf"}
Sep 16 19:00:38.520024 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.519755    2561 scope.go:117] "RemoveContainer" containerID="1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7"
Sep 16 19:00:38.527791 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.527775    2561 scope.go:117] "RemoveContainer" containerID="e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c"
Sep 16 19:00:38.534688 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.534672    2561 scope.go:117] "RemoveContainer" containerID="1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7"
Sep 16 19:00:38.534937 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:00:38.534919    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7\": container with ID starting with 1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7 not found: ID does not exist" containerID="1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7"
Sep 16 19:00:38.535005 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.534950    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7"} err="failed to get container status \"1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7\": rpc error: code = NotFound desc = could not find container \"1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7\": container with ID starting with 1201a51f579a32496c12141b1e241a4de78474ec78aa57e0519fa65db6b08cf7 not found: ID does not exist"
Sep 16 19:00:38.535005 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.534972    2561 scope.go:117] "RemoveContainer" containerID="e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c"
Sep 16 19:00:38.535223 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:00:38.535205    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c\": container with ID starting with e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c not found: ID does not exist" containerID="e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c"
Sep 16 19:00:38.535272 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.535230    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c"} err="failed to get container status \"e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c\": rpc error: code = NotFound desc = could not find container \"e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c\": container with ID starting with e2ba5d05317639a799988480359d6b7f26ffd5059caa06d4bc44c3dacde7bc3c not found: ID does not exist"
Sep 16 19:00:38.545390 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.545349    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"]
Sep 16 19:00:38.547076 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:38.547056    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-15f71-predictor-5c75fd5cf5-mz2q8"]
Sep 16 19:00:39.152056 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:39.152022    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" path="/var/lib/kubelet/pods/3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b/volumes"
Sep 16 19:00:42.519420 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:42.519391    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:00:42.519897 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:42.519872    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 16 19:00:52.520805 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:00:52.520763    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 16 19:01:02.520845 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:02.520804    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 16 19:01:12.519909 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:12.519863    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 16 19:01:22.520485 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:22.520408    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:01:24.263287 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.263254    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"]
Sep 16 19:01:24.263672 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.263494    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container" containerID="cri-o://1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a" gracePeriod=30
Sep 16 19:01:24.263672 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.263538    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kube-rbac-proxy" containerID="cri-o://643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65" gracePeriod=30
Sep 16 19:01:24.509898 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.509866    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"]
Sep 16 19:01:24.510131 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.510116    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container"
Sep 16 19:01:24.510131 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.510131    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container"
Sep 16 19:01:24.510228 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.510145    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kube-rbac-proxy"
Sep 16 19:01:24.510228 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.510150    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kube-rbac-proxy"
Sep 16 19:01:24.510228 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.510196    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kserve-container"
Sep 16 19:01:24.510228 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.510204    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a04d9ae-1ae6-46ca-a41d-d3f404f8aa6b" containerName="kube-rbac-proxy"
Sep 16 19:01:24.512977 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.512961    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:24.515988 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.515930    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-34033-predictor-serving-cert\""
Sep 16 19:01:24.515988 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.515964    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-34033-kube-rbac-proxy-sar-config\""
Sep 16 19:01:24.524431 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.524405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"]
Sep 16 19:01:24.597283 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.597247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deef9f80-1a1a-4bf4-9307-61322365601a-proxy-tls\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:24.597494 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.597285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxbjt\" (UniqueName: \"kubernetes.io/projected/deef9f80-1a1a-4bf4-9307-61322365601a-kube-api-access-gxbjt\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:24.597494 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.597336    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deef9f80-1a1a-4bf4-9307-61322365601a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:24.679079 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.679045    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerID="643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65" exitCode=2
Sep 16 19:01:24.679232 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.679083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" event={"ID":"ccddb524-6850-46d8-85f9-7dce31e6bec7","Type":"ContainerDied","Data":"643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65"}
Sep 16 19:01:24.698527 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.698498    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deef9f80-1a1a-4bf4-9307-61322365601a-proxy-tls\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:24.698642 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.698537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxbjt\" (UniqueName: \"kubernetes.io/projected/deef9f80-1a1a-4bf4-9307-61322365601a-kube-api-access-gxbjt\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:24.698642 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.698565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deef9f80-1a1a-4bf4-9307-61322365601a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:24.698749 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:01:24.698663    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-34033-predictor-serving-cert: secret "error-404-isvc-34033-predictor-serving-cert" not found
Sep 16 19:01:24.698749 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:01:24.698736    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deef9f80-1a1a-4bf4-9307-61322365601a-proxy-tls podName:deef9f80-1a1a-4bf4-9307-61322365601a nodeName:}" failed. No retries permitted until 2026-09-16 19:01:25.198716765 +0000 UTC m=+1256.553581386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/deef9f80-1a1a-4bf4-9307-61322365601a-proxy-tls") pod "error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" (UID: "deef9f80-1a1a-4bf4-9307-61322365601a") : secret "error-404-isvc-34033-predictor-serving-cert" not found
Sep 16 19:01:24.699218 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.699198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deef9f80-1a1a-4bf4-9307-61322365601a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:24.716608 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:24.716584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxbjt\" (UniqueName: \"kubernetes.io/projected/deef9f80-1a1a-4bf4-9307-61322365601a-kube-api-access-gxbjt\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:25.201779 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:25.201748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deef9f80-1a1a-4bf4-9307-61322365601a-proxy-tls\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:25.204090 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:25.204068    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deef9f80-1a1a-4bf4-9307-61322365601a-proxy-tls\") pod \"error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") " pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:25.422312 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:25.422272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:25.711214 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:25.711189    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"]
Sep 16 19:01:25.713534 ip-10-0-128-29 kubenswrapper[2561]: W0916 19:01:25.713507    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeef9f80_1a1a_4bf4_9307_61322365601a.slice/crio-53652d2749242e96002e51fc7eb6d137ff91bf2f3bf84dfd5c899e61bedb161e WatchSource:0}: Error finding container 53652d2749242e96002e51fc7eb6d137ff91bf2f3bf84dfd5c899e61bedb161e: Status 404 returned error can't find the container with id 53652d2749242e96002e51fc7eb6d137ff91bf2f3bf84dfd5c899e61bedb161e
Sep 16 19:01:26.688477 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:26.688433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" event={"ID":"deef9f80-1a1a-4bf4-9307-61322365601a","Type":"ContainerStarted","Data":"c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2"}
Sep 16 19:01:26.688864 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:26.688516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" event={"ID":"deef9f80-1a1a-4bf4-9307-61322365601a","Type":"ContainerStarted","Data":"6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0"}
Sep 16 19:01:26.688864 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:26.688536    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" event={"ID":"deef9f80-1a1a-4bf4-9307-61322365601a","Type":"ContainerStarted","Data":"53652d2749242e96002e51fc7eb6d137ff91bf2f3bf84dfd5c899e61bedb161e"}
Sep 16 19:01:26.688864 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:26.688566    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:26.714180 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:26.714127    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" podStartSLOduration=2.714109021 podStartE2EDuration="2.714109021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:26.71397541 +0000 UTC m=+1258.068840048" watchObservedRunningTime="2026-09-16 19:01:26.714109021 +0000 UTC m=+1258.068973661"
Sep 16 19:01:27.203513 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.203491    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 19:01:27.218626 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.218604    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb524-6850-46d8-85f9-7dce31e6bec7-proxy-tls\") pod \"ccddb524-6850-46d8-85f9-7dce31e6bec7\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") "
Sep 16 19:01:27.218743 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.218668    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b429f\" (UniqueName: \"kubernetes.io/projected/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-api-access-b429f\") pod \"ccddb524-6850-46d8-85f9-7dce31e6bec7\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") "
Sep 16 19:01:27.218801 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.218769    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-rbac-proxy-sar-config\") pod \"ccddb524-6850-46d8-85f9-7dce31e6bec7\" (UID: \"ccddb524-6850-46d8-85f9-7dce31e6bec7\") "
Sep 16 19:01:27.219088 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.219064    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ccddb524-6850-46d8-85f9-7dce31e6bec7" (UID: "ccddb524-6850-46d8-85f9-7dce31e6bec7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:27.220668 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.220651    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccddb524-6850-46d8-85f9-7dce31e6bec7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ccddb524-6850-46d8-85f9-7dce31e6bec7" (UID: "ccddb524-6850-46d8-85f9-7dce31e6bec7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:27.220741 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.220672    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-api-access-b429f" (OuterVolumeSpecName: "kube-api-access-b429f") pod "ccddb524-6850-46d8-85f9-7dce31e6bec7" (UID: "ccddb524-6850-46d8-85f9-7dce31e6bec7"). InnerVolumeSpecName "kube-api-access-b429f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:27.319359 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.319258    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b429f\" (UniqueName: \"kubernetes.io/projected/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-api-access-b429f\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:01:27.319359 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.319289    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ccddb524-6850-46d8-85f9-7dce31e6bec7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:01:27.319359 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.319317    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ccddb524-6850-46d8-85f9-7dce31e6bec7-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:01:27.694712 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.694616    2561 generic.go:356] "Generic (PLEG): container finished" podID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerID="1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a" exitCode=0
Sep 16 19:01:27.694712 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.694668    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"
Sep 16 19:01:27.694712 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.694702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" event={"ID":"ccddb524-6850-46d8-85f9-7dce31e6bec7","Type":"ContainerDied","Data":"1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a"}
Sep 16 19:01:27.695260 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.694773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9" event={"ID":"ccddb524-6850-46d8-85f9-7dce31e6bec7","Type":"ContainerDied","Data":"c97a9101f59e72c461d73f24222419389ff424c8ba41141d16f42d28a44c2011"}
Sep 16 19:01:27.695260 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.694815    2561 scope.go:117] "RemoveContainer" containerID="643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65"
Sep 16 19:01:27.695260 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.695193    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:27.696658 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.696636    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 16 19:01:27.703426 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.703409    2561 scope.go:117] "RemoveContainer" containerID="1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a"
Sep 16 19:01:27.710642 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.710620    2561 scope.go:117] "RemoveContainer" containerID="643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65"
Sep 16 19:01:27.710906 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:01:27.710889    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65\": container with ID starting with 643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65 not found: ID does not exist" containerID="643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65"
Sep 16 19:01:27.710948 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.710914    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65"} err="failed to get container status \"643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65\": rpc error: code = NotFound desc = could not find container \"643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65\": container with ID starting with 643a90989912e2f2b462354801be7f1c6a13cf3f05acdee307c37cc4cbb0ed65 not found: ID does not exist"
Sep 16 19:01:27.710948 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.710929    2561 scope.go:117] "RemoveContainer" containerID="1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a"
Sep 16 19:01:27.711184 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:01:27.711166    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a\": container with ID starting with 1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a not found: ID does not exist" containerID="1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a"
Sep 16 19:01:27.711235 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.711191    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a"} err="failed to get container status \"1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a\": rpc error: code = NotFound desc = could not find container \"1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a\": container with ID starting with 1548626edb101eb59d03fc007998deda92b4ac91fada2d17a0d089195aed867a not found: ID does not exist"
Sep 16 19:01:27.722895 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.722865    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"]
Sep 16 19:01:27.724928 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:27.724905    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21301-predictor-6f496df769-xkcv9"]
Sep 16 19:01:28.699093 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:28.699055    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 16 19:01:29.152981 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:29.152945    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" path="/var/lib/kubelet/pods/ccddb524-6850-46d8-85f9-7dce31e6bec7/volumes"
Sep 16 19:01:33.703557 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:33.703526    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:01:33.704026 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:33.703995    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 16 19:01:43.704029 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:43.703990    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 16 19:01:45.137590 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.137556    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"]
Sep 16 19:01:45.137976 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.137762    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container" containerID="cri-o://338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36" gracePeriod=30
Sep 16 19:01:45.137976 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.137801    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kube-rbac-proxy" containerID="cri-o://9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807" gracePeriod=30
Sep 16 19:01:45.435404 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.435330    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"]
Sep 16 19:01:45.435662 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.435646    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container"
Sep 16 19:01:45.435736 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.435666    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container"
Sep 16 19:01:45.435736 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.435690    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kube-rbac-proxy"
Sep 16 19:01:45.435736 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.435699    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kube-rbac-proxy"
Sep 16 19:01:45.435888 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.435776    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kserve-container"
Sep 16 19:01:45.435888 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.435790    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccddb524-6850-46d8-85f9-7dce31e6bec7" containerName="kube-rbac-proxy"
Sep 16 19:01:45.438655 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.438636    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.441678 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.441649    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e4920-predictor-serving-cert\""
Sep 16 19:01:45.441678 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.441659    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e4920-kube-rbac-proxy-sar-config\""
Sep 16 19:01:45.450815 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.450794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"]
Sep 16 19:01:45.551836 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.551803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krkbs\" (UniqueName: \"kubernetes.io/projected/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-api-access-krkbs\") pod \"error-404-isvc-e4920-predictor-75647c67f-x4zcq\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") " pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.551950 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.551849    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e4920-predictor-75647c67f-x4zcq\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") " pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.551950 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.551886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b42904b-6de8-44c8-9520-5bc059b054d0-proxy-tls\") pod \"error-404-isvc-e4920-predictor-75647c67f-x4zcq\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") " pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.653060 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.653028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krkbs\" (UniqueName: \"kubernetes.io/projected/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-api-access-krkbs\") pod \"error-404-isvc-e4920-predictor-75647c67f-x4zcq\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") " pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.653173 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.653073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e4920-predictor-75647c67f-x4zcq\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") " pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.653173 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.653117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b42904b-6de8-44c8-9520-5bc059b054d0-proxy-tls\") pod \"error-404-isvc-e4920-predictor-75647c67f-x4zcq\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") " pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.653768 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.653746    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e4920-predictor-75647c67f-x4zcq\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") " pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.655477 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.655457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b42904b-6de8-44c8-9520-5bc059b054d0-proxy-tls\") pod \"error-404-isvc-e4920-predictor-75647c67f-x4zcq\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") " pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.664837 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.664808    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krkbs\" (UniqueName: \"kubernetes.io/projected/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-api-access-krkbs\") pod \"error-404-isvc-e4920-predictor-75647c67f-x4zcq\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") " pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.750918 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.750860    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:45.761681 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.761659    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerID="9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807" exitCode=2
Sep 16 19:01:45.761764 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:45.761714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" event={"ID":"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3","Type":"ContainerDied","Data":"9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807"}
Sep 16 19:01:46.028137 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:46.028106    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"]
Sep 16 19:01:46.031102 ip-10-0-128-29 kubenswrapper[2561]: W0916 19:01:46.031075    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b42904b_6de8_44c8_9520_5bc059b054d0.slice/crio-25b9138fbb4624184575505996865b0b0a50536f4f2848c23bde38e7ea6a3583 WatchSource:0}: Error finding container 25b9138fbb4624184575505996865b0b0a50536f4f2848c23bde38e7ea6a3583: Status 404 returned error can't find the container with id 25b9138fbb4624184575505996865b0b0a50536f4f2848c23bde38e7ea6a3583
Sep 16 19:01:46.766747 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:46.766711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" event={"ID":"9b42904b-6de8-44c8-9520-5bc059b054d0","Type":"ContainerStarted","Data":"0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8"}
Sep 16 19:01:46.767133 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:46.766775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" event={"ID":"9b42904b-6de8-44c8-9520-5bc059b054d0","Type":"ContainerStarted","Data":"eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a"}
Sep 16 19:01:46.767133 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:46.766798    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" event={"ID":"9b42904b-6de8-44c8-9520-5bc059b054d0","Type":"ContainerStarted","Data":"25b9138fbb4624184575505996865b0b0a50536f4f2848c23bde38e7ea6a3583"}
Sep 16 19:01:46.767133 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:46.766824    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:46.767133 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:46.766866    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:46.768261 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:46.768234    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 16 19:01:46.793760 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:46.793720    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podStartSLOduration=1.793708599 podStartE2EDuration="1.793708599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:46.792452672 +0000 UTC m=+1278.147317310" watchObservedRunningTime="2026-09-16 19:01:46.793708599 +0000 UTC m=+1278.148573236"
Sep 16 19:01:47.515659 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:47.515620    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Sep 16 19:01:47.770886 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:47.770796    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 16 19:01:48.176699 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.176675    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:01:48.271947 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.271922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-proxy-tls\") pod \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") "
Sep 16 19:01:48.272056 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.271976    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-rbac-proxy-sar-config\") pod \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") "
Sep 16 19:01:48.272056 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.272002    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z2bv\" (UniqueName: \"kubernetes.io/projected/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-api-access-5z2bv\") pod \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\" (UID: \"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3\") "
Sep 16 19:01:48.272352 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.272328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" (UID: "ca4e37bb-749a-42dd-a32a-ef7f2445e1a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:48.273796 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.273774    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" (UID: "ca4e37bb-749a-42dd-a32a-ef7f2445e1a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:48.273987 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.273968    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-api-access-5z2bv" (OuterVolumeSpecName: "kube-api-access-5z2bv") pod "ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" (UID: "ca4e37bb-749a-42dd-a32a-ef7f2445e1a3"). InnerVolumeSpecName "kube-api-access-5z2bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:48.373015 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.372961    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.373015 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.372981    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.373015 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.372990    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z2bv\" (UniqueName: \"kubernetes.io/projected/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3-kube-api-access-5z2bv\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.775368 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.775333    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerID="338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36" exitCode=0
Sep 16 19:01:48.775723 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.775384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" event={"ID":"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3","Type":"ContainerDied","Data":"338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36"}
Sep 16 19:01:48.775723 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.775432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z" event={"ID":"ca4e37bb-749a-42dd-a32a-ef7f2445e1a3","Type":"ContainerDied","Data":"8e0551be842cdfb415b104a9772430d01d6444ccd2057306cc016f96c6c7698f"}
Sep 16 19:01:48.775723 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.775389    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"
Sep 16 19:01:48.775723 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.775461    2561 scope.go:117] "RemoveContainer" containerID="9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807"
Sep 16 19:01:48.783340 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.783317    2561 scope.go:117] "RemoveContainer" containerID="338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36"
Sep 16 19:01:48.790137 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.790117    2561 scope.go:117] "RemoveContainer" containerID="9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807"
Sep 16 19:01:48.790487 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:01:48.790465    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807\": container with ID starting with 9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807 not found: ID does not exist" containerID="9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807"
Sep 16 19:01:48.790579 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.790500    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807"} err="failed to get container status \"9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807\": rpc error: code = NotFound desc = could not find container \"9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807\": container with ID starting with 9171e1171d60ace5852578a60db70992a6b15d5baaef01972bb112db20f5e807 not found: ID does not exist"
Sep 16 19:01:48.790579 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.790522    2561 scope.go:117] "RemoveContainer" containerID="338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36"
Sep 16 19:01:48.790786 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:01:48.790769    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36\": container with ID starting with 338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36 not found: ID does not exist" containerID="338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36"
Sep 16 19:01:48.790826 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.790791    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36"} err="failed to get container status \"338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36\": rpc error: code = NotFound desc = could not find container \"338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36\": container with ID starting with 338b51d9fdfda033328ee6f141f744591f82b556dbb6f6d2e5243f9c1bcb8f36 not found: ID does not exist"
Sep 16 19:01:48.800113 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.800091    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"]
Sep 16 19:01:48.803752 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:48.803733    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1593f-predictor-599ddd5c95-jfd8z"]
Sep 16 19:01:49.152981 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:49.152905    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" path="/var/lib/kubelet/pods/ca4e37bb-749a-42dd-a32a-ef7f2445e1a3/volumes"
Sep 16 19:01:52.775413 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:52.775382    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:01:52.775870 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:52.775835    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 16 19:01:53.704083 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:01:53.704022    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 16 19:02:02.776668 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:02.776624    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 16 19:02:03.704599 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:03.704561    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 16 19:02:12.776360 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:12.776322    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 16 19:02:13.705223 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:13.705195    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:02:22.776471 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:22.776430    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 16 19:02:32.777188 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:32.777150    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:02:34.737521 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:34.737490    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"]
Sep 16 19:02:34.737963 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:34.737732    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container" containerID="cri-o://6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0" gracePeriod=30
Sep 16 19:02:34.737963 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:34.737774    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kube-rbac-proxy" containerID="cri-o://c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2" gracePeriod=30
Sep 16 19:02:34.940576 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:34.940548    2561 generic.go:356] "Generic (PLEG): container finished" podID="deef9f80-1a1a-4bf4-9307-61322365601a" containerID="c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2" exitCode=2
Sep 16 19:02:34.940677 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:34.940617    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" event={"ID":"deef9f80-1a1a-4bf4-9307-61322365601a","Type":"ContainerDied","Data":"c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2"}
Sep 16 19:02:35.168468 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.168433    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"]
Sep 16 19:02:35.168756 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.168738    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kube-rbac-proxy"
Sep 16 19:02:35.168834 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.168757    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kube-rbac-proxy"
Sep 16 19:02:35.168834 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.168784    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container"
Sep 16 19:02:35.168834 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.168792    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container"
Sep 16 19:02:35.168986 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.168858    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kserve-container"
Sep 16 19:02:35.168986 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.168870    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4e37bb-749a-42dd-a32a-ef7f2445e1a3" containerName="kube-rbac-proxy"
Sep 16 19:02:35.171726 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.171709    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.176838 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.176820    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e3b29-predictor-serving-cert\""
Sep 16 19:02:35.177163 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.177143    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e3b29-kube-rbac-proxy-sar-config\""
Sep 16 19:02:35.194239 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.194199    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"]
Sep 16 19:02:35.295745 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.295719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-proxy-tls\") pod \"error-404-isvc-e3b29-predictor-596b8c9d77-b79mj\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") " pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.295844 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.295760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e3b29-predictor-596b8c9d77-b79mj\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") " pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.295844 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.295798    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkllr\" (UniqueName: \"kubernetes.io/projected/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-api-access-vkllr\") pod \"error-404-isvc-e3b29-predictor-596b8c9d77-b79mj\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") " pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.396861 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.396836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkllr\" (UniqueName: \"kubernetes.io/projected/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-api-access-vkllr\") pod \"error-404-isvc-e3b29-predictor-596b8c9d77-b79mj\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") " pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.396957 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.396896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-proxy-tls\") pod \"error-404-isvc-e3b29-predictor-596b8c9d77-b79mj\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") " pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.396957 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.396918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e3b29-predictor-596b8c9d77-b79mj\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") " pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.397554 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.397536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e3b29-predictor-596b8c9d77-b79mj\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") " pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.399211 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.399189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-proxy-tls\") pod \"error-404-isvc-e3b29-predictor-596b8c9d77-b79mj\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") " pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.413268 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.413241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkllr\" (UniqueName: \"kubernetes.io/projected/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-api-access-vkllr\") pod \"error-404-isvc-e3b29-predictor-596b8c9d77-b79mj\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") " pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.480699 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.480650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.754820 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.754794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"]
Sep 16 19:02:35.756868 ip-10-0-128-29 kubenswrapper[2561]: W0916 19:02:35.756843    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ae97cb_5ae3_45f0_ba3f_c038a32f75a4.slice/crio-9d03d4efe231be6c7398fa9fd0163892ec01974cb09e6df3ef54b1ed1ca1fcc4 WatchSource:0}: Error finding container 9d03d4efe231be6c7398fa9fd0163892ec01974cb09e6df3ef54b1ed1ca1fcc4: Status 404 returned error can't find the container with id 9d03d4efe231be6c7398fa9fd0163892ec01974cb09e6df3ef54b1ed1ca1fcc4
Sep 16 19:02:35.947345 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.947315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" event={"ID":"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4","Type":"ContainerStarted","Data":"00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5"}
Sep 16 19:02:35.947452 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.947382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" event={"ID":"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4","Type":"ContainerStarted","Data":"0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f"}
Sep 16 19:02:35.947452 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.947409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" event={"ID":"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4","Type":"ContainerStarted","Data":"9d03d4efe231be6c7398fa9fd0163892ec01974cb09e6df3ef54b1ed1ca1fcc4"}
Sep 16 19:02:35.947452 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.947442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.947611 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.947468    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:35.948641 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.948616    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 16 19:02:35.978729 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:35.978680    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podStartSLOduration=0.97866205 podStartE2EDuration="978.66205ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:35.977966141 +0000 UTC m=+1327.332830779" watchObservedRunningTime="2026-09-16 19:02:35.97866205 +0000 UTC m=+1327.333526690"
Sep 16 19:02:36.952484 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:36.952444    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 16 19:02:37.480046 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.480023    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:02:37.610881 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.610855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deef9f80-1a1a-4bf4-9307-61322365601a-proxy-tls\") pod \"deef9f80-1a1a-4bf4-9307-61322365601a\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") "
Sep 16 19:02:37.610986 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.610891    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deef9f80-1a1a-4bf4-9307-61322365601a-kube-rbac-proxy-sar-config\") pod \"deef9f80-1a1a-4bf4-9307-61322365601a\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") "
Sep 16 19:02:37.610986 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.610951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxbjt\" (UniqueName: \"kubernetes.io/projected/deef9f80-1a1a-4bf4-9307-61322365601a-kube-api-access-gxbjt\") pod \"deef9f80-1a1a-4bf4-9307-61322365601a\" (UID: \"deef9f80-1a1a-4bf4-9307-61322365601a\") "
Sep 16 19:02:37.611254 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.611229    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deef9f80-1a1a-4bf4-9307-61322365601a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "deef9f80-1a1a-4bf4-9307-61322365601a" (UID: "deef9f80-1a1a-4bf4-9307-61322365601a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:37.612893 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.612869    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deef9f80-1a1a-4bf4-9307-61322365601a-kube-api-access-gxbjt" (OuterVolumeSpecName: "kube-api-access-gxbjt") pod "deef9f80-1a1a-4bf4-9307-61322365601a" (UID: "deef9f80-1a1a-4bf4-9307-61322365601a"). InnerVolumeSpecName "kube-api-access-gxbjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:37.612976 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.612909    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deef9f80-1a1a-4bf4-9307-61322365601a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "deef9f80-1a1a-4bf4-9307-61322365601a" (UID: "deef9f80-1a1a-4bf4-9307-61322365601a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:37.711839 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.711819    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxbjt\" (UniqueName: \"kubernetes.io/projected/deef9f80-1a1a-4bf4-9307-61322365601a-kube-api-access-gxbjt\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:02:37.711839 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.711838    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/deef9f80-1a1a-4bf4-9307-61322365601a-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:02:37.711958 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.711849    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/deef9f80-1a1a-4bf4-9307-61322365601a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:02:37.957796 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.957739    2561 generic.go:356] "Generic (PLEG): container finished" podID="deef9f80-1a1a-4bf4-9307-61322365601a" containerID="6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0" exitCode=0
Sep 16 19:02:37.957796 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.957793    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"
Sep 16 19:02:37.958108 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.957827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" event={"ID":"deef9f80-1a1a-4bf4-9307-61322365601a","Type":"ContainerDied","Data":"6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0"}
Sep 16 19:02:37.958108 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.957887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8" event={"ID":"deef9f80-1a1a-4bf4-9307-61322365601a","Type":"ContainerDied","Data":"53652d2749242e96002e51fc7eb6d137ff91bf2f3bf84dfd5c899e61bedb161e"}
Sep 16 19:02:37.958108 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.957916    2561 scope.go:117] "RemoveContainer" containerID="c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2"
Sep 16 19:02:37.966057 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.966035    2561 scope.go:117] "RemoveContainer" containerID="6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0"
Sep 16 19:02:37.972707 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.972693    2561 scope.go:117] "RemoveContainer" containerID="c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2"
Sep 16 19:02:37.972961 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:02:37.972944    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2\": container with ID starting with c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2 not found: ID does not exist" containerID="c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2"
Sep 16 19:02:37.973002 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.972969    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2"} err="failed to get container status \"c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2\": rpc error: code = NotFound desc = could not find container \"c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2\": container with ID starting with c11665823399e222c86d830574da632bc753e6d8fcd6b091514001eadcc4cea2 not found: ID does not exist"
Sep 16 19:02:37.973002 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.972985    2561 scope.go:117] "RemoveContainer" containerID="6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0"
Sep 16 19:02:37.973232 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:02:37.973215    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0\": container with ID starting with 6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0 not found: ID does not exist" containerID="6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0"
Sep 16 19:02:37.973267 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.973237    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0"} err="failed to get container status \"6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0\": rpc error: code = NotFound desc = could not find container \"6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0\": container with ID starting with 6ae2bfc84163b5193117b1134d7551b47d66591d9baee548074752ff71f3dce0 not found: ID does not exist"
Sep 16 19:02:37.986441 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.986424    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"]
Sep 16 19:02:37.991340 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:37.991320    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-34033-predictor-b4f8cfdcc-p2gj8"]
Sep 16 19:02:39.152763 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:39.152728    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" path="/var/lib/kubelet/pods/deef9f80-1a1a-4bf4-9307-61322365601a/volumes"
Sep 16 19:02:41.957943 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:41.957915    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:02:41.958426 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:41.958403    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 16 19:02:51.958370 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:02:51.958325    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 16 19:03:01.958761 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:03:01.958721    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 16 19:03:11.958453 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:03:11.958414    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 16 19:03:21.958955 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:03:21.958910    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:11:00.193469 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.193434    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"]
Sep 16 19:11:00.195931 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.193656    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container" containerID="cri-o://eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a" gracePeriod=30
Sep 16 19:11:00.195931 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.193706    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kube-rbac-proxy" containerID="cri-o://0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8" gracePeriod=30
Sep 16 19:11:00.645253 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.645218    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"]
Sep 16 19:11:00.645626 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.645605    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container"
Sep 16 19:11:00.645689 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.645630    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container"
Sep 16 19:11:00.645689 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.645650    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kube-rbac-proxy"
Sep 16 19:11:00.645689 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.645658    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kube-rbac-proxy"
Sep 16 19:11:00.645837 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.645715    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kube-rbac-proxy"
Sep 16 19:11:00.645837 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.645726    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="deef9f80-1a1a-4bf4-9307-61322365601a" containerName="kserve-container"
Sep 16 19:11:00.648868 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.648849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:00.651894 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.651876    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-617e5-predictor-serving-cert\""
Sep 16 19:11:00.653246 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.653229    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-617e5-kube-rbac-proxy-sar-config\""
Sep 16 19:11:00.695800 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.695769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:00.695933 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.695837    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxtql\" (UniqueName: \"kubernetes.io/projected/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-api-access-dxtql\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:00.695933 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.695913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-proxy-tls\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:00.717136 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.717111    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerID="0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8" exitCode=2
Sep 16 19:11:00.717237 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.717172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" event={"ID":"9b42904b-6de8-44c8-9520-5bc059b054d0","Type":"ContainerDied","Data":"0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8"}
Sep 16 19:11:00.755274 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.755249    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"]
Sep 16 19:11:00.796510 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.796483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:00.796613 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.796537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxtql\" (UniqueName: \"kubernetes.io/projected/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-api-access-dxtql\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:00.796613 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.796568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-proxy-tls\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:00.796727 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:11:00.796712    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-617e5-predictor-serving-cert: secret "error-404-isvc-617e5-predictor-serving-cert" not found
Sep 16 19:11:00.796796 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:11:00.796784    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-proxy-tls podName:9604a92a-ce44-4c0c-b3d4-4c00fa89281b nodeName:}" failed. No retries permitted until 2026-09-16 19:11:01.296764671 +0000 UTC m=+1832.651629291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-proxy-tls") pod "error-404-isvc-617e5-predictor-6dd67cff67-zcf86" (UID: "9604a92a-ce44-4c0c-b3d4-4c00fa89281b") : secret "error-404-isvc-617e5-predictor-serving-cert" not found
Sep 16 19:11:00.797210 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.797191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:00.821404 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:00.821378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxtql\" (UniqueName: \"kubernetes.io/projected/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-api-access-dxtql\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:01.299429 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:01.299390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-proxy-tls\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:01.301755 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:01.301730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-proxy-tls\") pod \"error-404-isvc-617e5-predictor-6dd67cff67-zcf86\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") " pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:01.558488 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:01.558399    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:01.852215 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:01.852190    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"]
Sep 16 19:11:01.854125 ip-10-0-128-29 kubenswrapper[2561]: W0916 19:11:01.854090    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9604a92a_ce44_4c0c_b3d4_4c00fa89281b.slice/crio-adb8c63efe84299eb4da6d84b47d5a82ac71cae48a172de73e9a70811547f9d8 WatchSource:0}: Error finding container adb8c63efe84299eb4da6d84b47d5a82ac71cae48a172de73e9a70811547f9d8: Status 404 returned error can't find the container with id adb8c63efe84299eb4da6d84b47d5a82ac71cae48a172de73e9a70811547f9d8
Sep 16 19:11:02.727088 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:02.727057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" event={"ID":"9604a92a-ce44-4c0c-b3d4-4c00fa89281b","Type":"ContainerStarted","Data":"0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde"}
Sep 16 19:11:02.727554 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:02.727113    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:02.727554 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:02.727238    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:02.727554 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:02.727280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" event={"ID":"9604a92a-ce44-4c0c-b3d4-4c00fa89281b","Type":"ContainerStarted","Data":"623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260"}
Sep 16 19:11:02.727554 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:02.727314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" event={"ID":"9604a92a-ce44-4c0c-b3d4-4c00fa89281b","Type":"ContainerStarted","Data":"adb8c63efe84299eb4da6d84b47d5a82ac71cae48a172de73e9a70811547f9d8"}
Sep 16 19:11:02.728342 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:02.728320    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 19:11:02.770983 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:02.770945    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 16 19:11:02.776629 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:02.776603    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 16 19:11:03.352651 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.352626    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:11:03.385013 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.384968    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" podStartSLOduration=3.384954369 podStartE2EDuration="3.384954369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:02.77224464 +0000 UTC m=+1834.127109292" watchObservedRunningTime="2026-09-16 19:11:03.384954369 +0000 UTC m=+1834.739819007"
Sep 16 19:11:03.412505 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.412451    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-rbac-proxy-sar-config\") pod \"9b42904b-6de8-44c8-9520-5bc059b054d0\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") "
Sep 16 19:11:03.412505 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.412486    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krkbs\" (UniqueName: \"kubernetes.io/projected/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-api-access-krkbs\") pod \"9b42904b-6de8-44c8-9520-5bc059b054d0\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") "
Sep 16 19:11:03.412505 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.412504    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b42904b-6de8-44c8-9520-5bc059b054d0-proxy-tls\") pod \"9b42904b-6de8-44c8-9520-5bc059b054d0\" (UID: \"9b42904b-6de8-44c8-9520-5bc059b054d0\") "
Sep 16 19:11:03.412786 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.412762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b42904b-6de8-44c8-9520-5bc059b054d0" (UID: "9b42904b-6de8-44c8-9520-5bc059b054d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:03.414499 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.414470    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b42904b-6de8-44c8-9520-5bc059b054d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b42904b-6de8-44c8-9520-5bc059b054d0" (UID: "9b42904b-6de8-44c8-9520-5bc059b054d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:03.414587 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.414557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-api-access-krkbs" (OuterVolumeSpecName: "kube-api-access-krkbs") pod "9b42904b-6de8-44c8-9520-5bc059b054d0" (UID: "9b42904b-6de8-44c8-9520-5bc059b054d0"). InnerVolumeSpecName "kube-api-access-krkbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:03.513478 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.513443    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:11:03.513478 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.513476    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krkbs\" (UniqueName: \"kubernetes.io/projected/9b42904b-6de8-44c8-9520-5bc059b054d0-kube-api-access-krkbs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:11:03.513478 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.513487    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b42904b-6de8-44c8-9520-5bc059b054d0-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:11:03.733132 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.733035    2561 generic.go:356] "Generic (PLEG): container finished" podID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerID="eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a" exitCode=0
Sep 16 19:11:03.733132 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.733091    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"
Sep 16 19:11:03.733647 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.733123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" event={"ID":"9b42904b-6de8-44c8-9520-5bc059b054d0","Type":"ContainerDied","Data":"eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a"}
Sep 16 19:11:03.733647 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.733197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq" event={"ID":"9b42904b-6de8-44c8-9520-5bc059b054d0","Type":"ContainerDied","Data":"25b9138fbb4624184575505996865b0b0a50536f4f2848c23bde38e7ea6a3583"}
Sep 16 19:11:03.733647 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.733241    2561 scope.go:117] "RemoveContainer" containerID="0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8"
Sep 16 19:11:03.733853 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.733827    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 19:11:03.742153 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.742135    2561 scope.go:117] "RemoveContainer" containerID="eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a"
Sep 16 19:11:03.750543 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.750519    2561 scope.go:117] "RemoveContainer" containerID="0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8"
Sep 16 19:11:03.750791 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:11:03.750775    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8\": container with ID starting with 0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8 not found: ID does not exist" containerID="0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8"
Sep 16 19:11:03.750844 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.750797    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8"} err="failed to get container status \"0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8\": rpc error: code = NotFound desc = could not find container \"0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8\": container with ID starting with 0b927aa2ac551517f3bb82920fb1c3ec2f7b2b5d7330bf11d7c9e978dcca12d8 not found: ID does not exist"
Sep 16 19:11:03.750844 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.750811    2561 scope.go:117] "RemoveContainer" containerID="eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a"
Sep 16 19:11:03.751098 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:11:03.751080    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a\": container with ID starting with eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a not found: ID does not exist" containerID="eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a"
Sep 16 19:11:03.751150 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.751106    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a"} err="failed to get container status \"eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a\": rpc error: code = NotFound desc = could not find container \"eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a\": container with ID starting with eb244a3e53d44a1a2d9bcce4af0d1e8e4b001d2fdca9d6f3d12d045eafaa4f2a not found: ID does not exist"
Sep 16 19:11:03.765348 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.765322    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"]
Sep 16 19:11:03.770452 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:03.770431    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4920-predictor-75647c67f-x4zcq"]
Sep 16 19:11:05.151826 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:05.151794    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" path="/var/lib/kubelet/pods/9b42904b-6de8-44c8-9520-5bc059b054d0/volumes"
Sep 16 19:11:08.738053 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:08.738025    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:08.738659 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:08.738625    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 19:11:18.739217 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:18.739162    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 19:11:28.738844 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:28.738792    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 19:11:38.739521 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:38.739477    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 16 19:11:48.739450 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:48.739413    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:11:49.644252 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.644220    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"]
Sep 16 19:11:49.644491 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.644439    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container" containerID="cri-o://0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f" gracePeriod=30
Sep 16 19:11:49.644645 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.644482    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kube-rbac-proxy" containerID="cri-o://00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5" gracePeriod=30
Sep 16 19:11:49.902218 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.902142    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerID="00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5" exitCode=2
Sep 16 19:11:49.902218 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.902196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" event={"ID":"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4","Type":"ContainerDied","Data":"00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5"}
Sep 16 19:11:49.950253 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.950228    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"]
Sep 16 19:11:49.950561 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.950545    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kube-rbac-proxy"
Sep 16 19:11:49.950637 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.950564    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kube-rbac-proxy"
Sep 16 19:11:49.950637 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.950585    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container"
Sep 16 19:11:49.950637 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.950594    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container"
Sep 16 19:11:49.950789 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.950689    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kube-rbac-proxy"
Sep 16 19:11:49.950789 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.950704    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b42904b-6de8-44c8-9520-5bc059b054d0" containerName="kserve-container"
Sep 16 19:11:49.953658 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.953640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:49.956699 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.956666    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d2d49-predictor-serving-cert\""
Sep 16 19:11:49.956785 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:49.956744    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d2d49-kube-rbac-proxy-sar-config\""
Sep 16 19:11:50.031352 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.031321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4zc\" (UniqueName: \"kubernetes.io/projected/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-api-access-lm4zc\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.031479 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.031371    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-proxy-tls\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.031479 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.031414    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.050230 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.050202    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"]
Sep 16 19:11:50.132249 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.132224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4zc\" (UniqueName: \"kubernetes.io/projected/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-api-access-lm4zc\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.132393 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.132260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-proxy-tls\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.132393 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.132282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.132393 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:11:50.132382    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-serving-cert: secret "error-404-isvc-d2d49-predictor-serving-cert" not found
Sep 16 19:11:50.132553 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:11:50.132457    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-proxy-tls podName:bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:50.632436027 +0000 UTC m=+1881.987300646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-proxy-tls") pod "error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" (UID: "bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9") : secret "error-404-isvc-d2d49-predictor-serving-cert" not found
Sep 16 19:11:50.132879 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.132862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.147170 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.147149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4zc\" (UniqueName: \"kubernetes.io/projected/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-api-access-lm4zc\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.635877 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.635843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-proxy-tls\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.638210 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.638190    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-proxy-tls\") pod \"error-404-isvc-d2d49-predictor-9686f48b9-cnvdw\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") " pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:50.863727 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:50.863691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:51.142687 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:51.142665    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"]
Sep 16 19:11:51.143914 ip-10-0-128-29 kubenswrapper[2561]: W0916 19:11:51.143888    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb15c3d_6a36_4c0e_8b85_9a19c54be6e9.slice/crio-120dfc638fd001f397a4786d39d4886e610dbad9cff79d0d157c70acc8749287 WatchSource:0}: Error finding container 120dfc638fd001f397a4786d39d4886e610dbad9cff79d0d157c70acc8749287: Status 404 returned error can't find the container with id 120dfc638fd001f397a4786d39d4886e610dbad9cff79d0d157c70acc8749287
Sep 16 19:11:51.912231 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:51.912192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" event={"ID":"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9","Type":"ContainerStarted","Data":"6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e"}
Sep 16 19:11:51.912440 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:51.912249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" event={"ID":"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9","Type":"ContainerStarted","Data":"518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551"}
Sep 16 19:11:51.912440 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:51.912270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" event={"ID":"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9","Type":"ContainerStarted","Data":"120dfc638fd001f397a4786d39d4886e610dbad9cff79d0d157c70acc8749287"}
Sep 16 19:11:51.912440 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:51.912416    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:51.912575 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:51.912470    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:51.913767 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:51.913741    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 19:11:51.952685 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:51.952647    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 16 19:11:51.958400 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:51.958362    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 16 19:11:52.592985 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.592960    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:11:52.620191 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.620144    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" podStartSLOduration=3.620127652 podStartE2EDuration="3.620127652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:51.942826178 +0000 UTC m=+1883.297690827" watchObservedRunningTime="2026-09-16 19:11:52.620127652 +0000 UTC m=+1883.974992290"
Sep 16 19:11:52.754491 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.754405    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkllr\" (UniqueName: \"kubernetes.io/projected/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-api-access-vkllr\") pod \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") "
Sep 16 19:11:52.754491 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.754451    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-proxy-tls\") pod \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") "
Sep 16 19:11:52.754491 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.754478    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-rbac-proxy-sar-config\") pod \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\" (UID: \"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4\") "
Sep 16 19:11:52.754884 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.754853    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" (UID: "b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:52.756595 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.756574    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" (UID: "b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:52.756595 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.756586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-api-access-vkllr" (OuterVolumeSpecName: "kube-api-access-vkllr") pod "b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" (UID: "b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4"). InnerVolumeSpecName "kube-api-access-vkllr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:52.855930 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.855908    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkllr\" (UniqueName: \"kubernetes.io/projected/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-api-access-vkllr\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:11:52.855930 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.855928    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:11:52.856073 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.855939    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:11:52.917079 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.917055    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerID="0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f" exitCode=0
Sep 16 19:11:52.917179 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.917108    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"
Sep 16 19:11:52.917179 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.917142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" event={"ID":"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4","Type":"ContainerDied","Data":"0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f"}
Sep 16 19:11:52.917250 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.917225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj" event={"ID":"b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4","Type":"ContainerDied","Data":"9d03d4efe231be6c7398fa9fd0163892ec01974cb09e6df3ef54b1ed1ca1fcc4"}
Sep 16 19:11:52.917283 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.917260    2561 scope.go:117] "RemoveContainer" containerID="00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5"
Sep 16 19:11:52.917923 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.917896    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 19:11:52.925779 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.925765    2561 scope.go:117] "RemoveContainer" containerID="0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f"
Sep 16 19:11:52.932771 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.932754    2561 scope.go:117] "RemoveContainer" containerID="00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5"
Sep 16 19:11:52.933036 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:11:52.933020    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5\": container with ID starting with 00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5 not found: ID does not exist" containerID="00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5"
Sep 16 19:11:52.933078 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.933066    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5"} err="failed to get container status \"00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5\": rpc error: code = NotFound desc = could not find container \"00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5\": container with ID starting with 00eccf8c63a13e891de2e91e4c16cac053375b7b4fd6abe3d6ba82d99519d1f5 not found: ID does not exist"
Sep 16 19:11:52.933116 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.933083    2561 scope.go:117] "RemoveContainer" containerID="0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f"
Sep 16 19:11:52.933343 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:11:52.933327    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f\": container with ID starting with 0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f not found: ID does not exist" containerID="0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f"
Sep 16 19:11:52.933388 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.933350    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f"} err="failed to get container status \"0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f\": rpc error: code = NotFound desc = could not find container \"0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f\": container with ID starting with 0636b3c94ff2269b8262d3c3a101eff0365628ef89addf8fca16838ace09eb8f not found: ID does not exist"
Sep 16 19:11:52.949130 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.946487    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"]
Sep 16 19:11:52.962690 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:52.962669    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e3b29-predictor-596b8c9d77-b79mj"]
Sep 16 19:11:53.151418 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:53.151392    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" path="/var/lib/kubelet/pods/b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4/volumes"
Sep 16 19:11:57.921905 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:57.921879    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:11:57.922426 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:11:57.922398    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 19:12:07.923285 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:07.923242    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 19:12:10.950117 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:10.950081    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"]
Sep 16 19:12:10.950536 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:10.950322    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container" containerID="cri-o://623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260" gracePeriod=30
Sep 16 19:12:10.950536 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:10.950363    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kube-rbac-proxy" containerID="cri-o://0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde" gracePeriod=30
Sep 16 19:12:11.542702 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.538803    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"]
Sep 16 19:12:11.542702 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.539395    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container"
Sep 16 19:12:11.542702 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.539413    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container"
Sep 16 19:12:11.542702 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.539442    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kube-rbac-proxy"
Sep 16 19:12:11.542702 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.539451    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kube-rbac-proxy"
Sep 16 19:12:11.542702 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.539592    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kserve-container"
Sep 16 19:12:11.542702 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.539606    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ae97cb-5ae3-45f0-ba3f-c038a32f75a4" containerName="kube-rbac-proxy"
Sep 16 19:12:11.544006 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.543978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.551289 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.551261    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0e2c5-predictor-serving-cert\""
Sep 16 19:12:11.551419 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.551282    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0e2c5-kube-rbac-proxy-sar-config\""
Sep 16 19:12:11.681355 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.681323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8205d141-2120-49b0-bf31-f5c5c1156e5d-proxy-tls\") pod \"error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") " pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.681479 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.681366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") " pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.681479 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.681446    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hllv\" (UniqueName: \"kubernetes.io/projected/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-api-access-6hllv\") pod \"error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") " pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.691658 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.691630    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"]
Sep 16 19:12:11.782398 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.782368    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hllv\" (UniqueName: \"kubernetes.io/projected/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-api-access-6hllv\") pod \"error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") " pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.782521 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.782413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8205d141-2120-49b0-bf31-f5c5c1156e5d-proxy-tls\") pod \"error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") " pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.782521 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.782437    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") " pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.783003 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.782985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") " pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.784752 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.784730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8205d141-2120-49b0-bf31-f5c5c1156e5d-proxy-tls\") pod \"error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") " pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.793813 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.793759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hllv\" (UniqueName: \"kubernetes.io/projected/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-api-access-6hllv\") pod \"error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") " pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.854543 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.854521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:11.985164 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.985131    2561 generic.go:356] "Generic (PLEG): container finished" podID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerID="0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde" exitCode=2
Sep 16 19:12:11.985487 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:11.985203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" event={"ID":"9604a92a-ce44-4c0c-b3d4-4c00fa89281b","Type":"ContainerDied","Data":"0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde"}
Sep 16 19:12:12.146823 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:12.146753    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"]
Sep 16 19:12:12.150377 ip-10-0-128-29 kubenswrapper[2561]: W0916 19:12:12.150345    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8205d141_2120_49b0_bf31_f5c5c1156e5d.slice/crio-11c83034b50f75ae26cf2e55c01aba423f1bcabf5a250597b4c90be55f7366e9 WatchSource:0}: Error finding container 11c83034b50f75ae26cf2e55c01aba423f1bcabf5a250597b4c90be55f7366e9: Status 404 returned error can't find the container with id 11c83034b50f75ae26cf2e55c01aba423f1bcabf5a250597b4c90be55f7366e9
Sep 16 19:12:12.990874 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:12.990839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" event={"ID":"8205d141-2120-49b0-bf31-f5c5c1156e5d","Type":"ContainerStarted","Data":"1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a"}
Sep 16 19:12:12.991245 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:12.990892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" event={"ID":"8205d141-2120-49b0-bf31-f5c5c1156e5d","Type":"ContainerStarted","Data":"f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d"}
Sep 16 19:12:12.991245 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:12.990912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" event={"ID":"8205d141-2120-49b0-bf31-f5c5c1156e5d","Type":"ContainerStarted","Data":"11c83034b50f75ae26cf2e55c01aba423f1bcabf5a250597b4c90be55f7366e9"}
Sep 16 19:12:12.991245 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:12.990950    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:13.021592 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.021548    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" podStartSLOduration=2.021536997 podStartE2EDuration="2.021536997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:13.01986325 +0000 UTC m=+1904.374727887" watchObservedRunningTime="2026-09-16 19:12:13.021536997 +0000 UTC m=+1904.376401688"
Sep 16 19:12:13.698620 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.698596    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:12:13.798190 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.798168    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxtql\" (UniqueName: \"kubernetes.io/projected/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-api-access-dxtql\") pod \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") "
Sep 16 19:12:13.798337 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.798202    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-rbac-proxy-sar-config\") pod \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") "
Sep 16 19:12:13.798337 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.798223    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-proxy-tls\") pod \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\" (UID: \"9604a92a-ce44-4c0c-b3d4-4c00fa89281b\") "
Sep 16 19:12:13.798596 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.798568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9604a92a-ce44-4c0c-b3d4-4c00fa89281b" (UID: "9604a92a-ce44-4c0c-b3d4-4c00fa89281b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:13.800192 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.800170    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-api-access-dxtql" (OuterVolumeSpecName: "kube-api-access-dxtql") pod "9604a92a-ce44-4c0c-b3d4-4c00fa89281b" (UID: "9604a92a-ce44-4c0c-b3d4-4c00fa89281b"). InnerVolumeSpecName "kube-api-access-dxtql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:13.800267 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.800201    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9604a92a-ce44-4c0c-b3d4-4c00fa89281b" (UID: "9604a92a-ce44-4c0c-b3d4-4c00fa89281b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:13.899552 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.899529    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxtql\" (UniqueName: \"kubernetes.io/projected/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-api-access-dxtql\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:12:13.899552 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.899551    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:12:13.899674 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.899561    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9604a92a-ce44-4c0c-b3d4-4c00fa89281b-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:12:13.995973 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.995943    2561 generic.go:356] "Generic (PLEG): container finished" podID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerID="623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260" exitCode=0
Sep 16 19:12:13.996334 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.995999    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"
Sep 16 19:12:13.996334 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.996027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" event={"ID":"9604a92a-ce44-4c0c-b3d4-4c00fa89281b","Type":"ContainerDied","Data":"623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260"}
Sep 16 19:12:13.996334 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.996094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86" event={"ID":"9604a92a-ce44-4c0c-b3d4-4c00fa89281b","Type":"ContainerDied","Data":"adb8c63efe84299eb4da6d84b47d5a82ac71cae48a172de73e9a70811547f9d8"}
Sep 16 19:12:13.996334 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.996124    2561 scope.go:117] "RemoveContainer" containerID="0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde"
Sep 16 19:12:13.996553 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.996406    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:13.997664 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:13.997642    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 19:12:14.004949 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:14.004932    2561 scope.go:117] "RemoveContainer" containerID="623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260"
Sep 16 19:12:14.011926 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:14.011910    2561 scope.go:117] "RemoveContainer" containerID="0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde"
Sep 16 19:12:14.012163 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:12:14.012145    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde\": container with ID starting with 0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde not found: ID does not exist" containerID="0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde"
Sep 16 19:12:14.012214 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:14.012176    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde"} err="failed to get container status \"0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde\": rpc error: code = NotFound desc = could not find container \"0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde\": container with ID starting with 0e493d8d7cdb2174178dc1523fb0b98dcf4ef0b69d4391035d867b466e8fecde not found: ID does not exist"
Sep 16 19:12:14.012214 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:14.012196    2561 scope.go:117] "RemoveContainer" containerID="623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260"
Sep 16 19:12:14.012434 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:12:14.012415    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260\": container with ID starting with 623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260 not found: ID does not exist" containerID="623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260"
Sep 16 19:12:14.012485 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:14.012438    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260"} err="failed to get container status \"623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260\": rpc error: code = NotFound desc = could not find container \"623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260\": container with ID starting with 623e3992ab169d4bde16586722d2702338b5b1867f5f38a2018936d322995260 not found: ID does not exist"
Sep 16 19:12:14.030447 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:14.030425    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"]
Sep 16 19:12:14.041873 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:14.041854    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-617e5-predictor-6dd67cff67-zcf86"]
Sep 16 19:12:15.000175 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:15.000137    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 19:12:15.152363 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:15.152328    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" path="/var/lib/kubelet/pods/9604a92a-ce44-4c0c-b3d4-4c00fa89281b/volumes"
Sep 16 19:12:17.923035 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:17.922991    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 19:12:20.003954 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:20.003927    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:12:20.004566 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:20.004540    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 19:12:27.922382 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:27.922347    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 16 19:12:30.004982 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:30.004948    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 19:12:37.923436 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:37.923402    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:12:40.005423 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:40.005385    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 19:12:50.004462 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:12:50.004426    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 16 19:13:00.005879 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:13:00.005590    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:21:25.873591 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:25.873560    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"]
Sep 16 19:21:25.874102 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:25.873786    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container" containerID="cri-o://f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d" gracePeriod=30
Sep 16 19:21:25.874102 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:25.873829    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kube-rbac-proxy" containerID="cri-o://1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a" gracePeriod=30
Sep 16 19:21:26.902383 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:26.902345    2561 generic.go:356] "Generic (PLEG): container finished" podID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerID="1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a" exitCode=2
Sep 16 19:21:26.902738 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:26.902417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" event={"ID":"8205d141-2120-49b0-bf31-f5c5c1156e5d","Type":"ContainerDied","Data":"1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a"}
Sep 16 19:21:28.313136 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.313111    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:21:28.439761 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.439700    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hllv\" (UniqueName: \"kubernetes.io/projected/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-api-access-6hllv\") pod \"8205d141-2120-49b0-bf31-f5c5c1156e5d\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") "
Sep 16 19:21:28.439761 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.439737    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8205d141-2120-49b0-bf31-f5c5c1156e5d-proxy-tls\") pod \"8205d141-2120-49b0-bf31-f5c5c1156e5d\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") "
Sep 16 19:21:28.439761 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.439760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-rbac-proxy-sar-config\") pod \"8205d141-2120-49b0-bf31-f5c5c1156e5d\" (UID: \"8205d141-2120-49b0-bf31-f5c5c1156e5d\") "
Sep 16 19:21:28.440126 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.440101    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8205d141-2120-49b0-bf31-f5c5c1156e5d" (UID: "8205d141-2120-49b0-bf31-f5c5c1156e5d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:28.441743 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.441719    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8205d141-2120-49b0-bf31-f5c5c1156e5d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8205d141-2120-49b0-bf31-f5c5c1156e5d" (UID: "8205d141-2120-49b0-bf31-f5c5c1156e5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:28.441862 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.441844    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-api-access-6hllv" (OuterVolumeSpecName: "kube-api-access-6hllv") pod "8205d141-2120-49b0-bf31-f5c5c1156e5d" (UID: "8205d141-2120-49b0-bf31-f5c5c1156e5d"). InnerVolumeSpecName "kube-api-access-6hllv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:28.540621 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.540578    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hllv\" (UniqueName: \"kubernetes.io/projected/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-api-access-6hllv\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:21:28.540621 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.540615    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8205d141-2120-49b0-bf31-f5c5c1156e5d-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:21:28.540621 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.540627    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8205d141-2120-49b0-bf31-f5c5c1156e5d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:21:28.911555 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.911518    2561 generic.go:356] "Generic (PLEG): container finished" podID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerID="f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d" exitCode=0
Sep 16 19:21:28.911724 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.911568    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"
Sep 16 19:21:28.911724 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.911586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" event={"ID":"8205d141-2120-49b0-bf31-f5c5c1156e5d","Type":"ContainerDied","Data":"f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d"}
Sep 16 19:21:28.911724 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.911644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl" event={"ID":"8205d141-2120-49b0-bf31-f5c5c1156e5d","Type":"ContainerDied","Data":"11c83034b50f75ae26cf2e55c01aba423f1bcabf5a250597b4c90be55f7366e9"}
Sep 16 19:21:28.911724 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.911679    2561 scope.go:117] "RemoveContainer" containerID="1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a"
Sep 16 19:21:28.919704 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.919683    2561 scope.go:117] "RemoveContainer" containerID="f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d"
Sep 16 19:21:28.926458 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.926437    2561 scope.go:117] "RemoveContainer" containerID="1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a"
Sep 16 19:21:28.926717 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:21:28.926697    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a\": container with ID starting with 1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a not found: ID does not exist" containerID="1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a"
Sep 16 19:21:28.926762 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.926726    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a"} err="failed to get container status \"1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a\": rpc error: code = NotFound desc = could not find container \"1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a\": container with ID starting with 1da348211a0a9def6135f0ec9ffe76af087bce2f49586bcd21e2f21d85f3901a not found: ID does not exist"
Sep 16 19:21:28.926762 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.926742    2561 scope.go:117] "RemoveContainer" containerID="f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d"
Sep 16 19:21:28.926964 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:21:28.926949    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d\": container with ID starting with f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d not found: ID does not exist" containerID="f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d"
Sep 16 19:21:28.927008 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.926968    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d"} err="failed to get container status \"f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d\": rpc error: code = NotFound desc = could not find container \"f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d\": container with ID starting with f6ef14847dd310d9a5efb78b84d71520d0c8718b364e20f03cb08042f975fc8d not found: ID does not exist"
Sep 16 19:21:28.945030 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.945010    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"]
Sep 16 19:21:28.953542 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:28.953523    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0e2c5-predictor-7cb46d9cfc-7f2gl"]
Sep 16 19:21:29.152072 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:21:29.152041    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" path="/var/lib/kubelet/pods/8205d141-2120-49b0-bf31-f5c5c1156e5d/volumes"
Sep 16 19:29:09.343373 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:09.343340    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"]
Sep 16 19:29:09.345843 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:09.343622    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container" containerID="cri-o://518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551" gracePeriod=30
Sep 16 19:29:09.345843 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:09.343690    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kube-rbac-proxy" containerID="cri-o://6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e" gracePeriod=30
Sep 16 19:29:10.450684 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:10.450648    2561 generic.go:356] "Generic (PLEG): container finished" podID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerID="6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e" exitCode=2
Sep 16 19:29:10.451044 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:10.450719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" event={"ID":"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9","Type":"ContainerDied","Data":"6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e"}
Sep 16 19:29:11.584598 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.584578    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:29:11.723071 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.723012    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm4zc\" (UniqueName: \"kubernetes.io/projected/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-api-access-lm4zc\") pod \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") "
Sep 16 19:29:11.723071 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.723056    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-rbac-proxy-sar-config\") pod \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") "
Sep 16 19:29:11.723249 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.723088    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-proxy-tls\") pod \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\" (UID: \"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9\") "
Sep 16 19:29:11.723479 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.723452    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" (UID: "bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:11.725060 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.725033    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-api-access-lm4zc" (OuterVolumeSpecName: "kube-api-access-lm4zc") pod "bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" (UID: "bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9"). InnerVolumeSpecName "kube-api-access-lm4zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:11.725150 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.725107    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" (UID: "bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:11.823881 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.823855    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm4zc\" (UniqueName: \"kubernetes.io/projected/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-api-access-lm4zc\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:29:11.823881 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.823877    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:29:11.824002 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:11.823886    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9-proxy-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 16 19:29:12.460557 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.460528    2561 generic.go:356] "Generic (PLEG): container finished" podID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerID="518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551" exitCode=0
Sep 16 19:29:12.460685 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.460572    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"
Sep 16 19:29:12.460685 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.460596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" event={"ID":"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9","Type":"ContainerDied","Data":"518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551"}
Sep 16 19:29:12.460685 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.460650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw" event={"ID":"bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9","Type":"ContainerDied","Data":"120dfc638fd001f397a4786d39d4886e610dbad9cff79d0d157c70acc8749287"}
Sep 16 19:29:12.460685 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.460678    2561 scope.go:117] "RemoveContainer" containerID="6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e"
Sep 16 19:29:12.469712 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.469691    2561 scope.go:117] "RemoveContainer" containerID="518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551"
Sep 16 19:29:12.477351 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.477330    2561 scope.go:117] "RemoveContainer" containerID="6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e"
Sep 16 19:29:12.477696 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:29:12.477669    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e\": container with ID starting with 6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e not found: ID does not exist" containerID="6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e"
Sep 16 19:29:12.477768 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.477705    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e"} err="failed to get container status \"6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e\": rpc error: code = NotFound desc = could not find container \"6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e\": container with ID starting with 6653bd5b661e2bdfc772b479de8fb0e12bf5ac5c12a6170da0e1a04e2b177f6e not found: ID does not exist"
Sep 16 19:29:12.477768 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.477725    2561 scope.go:117] "RemoveContainer" containerID="518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551"
Sep 16 19:29:12.478037 ip-10-0-128-29 kubenswrapper[2561]: E0916 19:29:12.478016    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551\": container with ID starting with 518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551 not found: ID does not exist" containerID="518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551"
Sep 16 19:29:12.478081 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.478042    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551"} err="failed to get container status \"518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551\": rpc error: code = NotFound desc = could not find container \"518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551\": container with ID starting with 518ea48845f5619e3e678193d17b50073049cbdd35f9b893e59ef555ba6b0551 not found: ID does not exist"
Sep 16 19:29:12.505892 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.505866    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"]
Sep 16 19:29:12.511763 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:12.511736    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2d49-predictor-9686f48b9-cnvdw"]
Sep 16 19:29:13.156149 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:13.156122    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" path="/var/lib/kubelet/pods/bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9/volumes"
Sep 16 19:29:44.311098 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:44.311064    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-96kgg_012ca71e-d6e0-45af-a910-83d05cc0dd6b/global-pull-secret-syncer/0.log"
Sep 16 19:29:44.440896 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:44.440863    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vslg4_13da0f1f-abdc-4b10-a33d-25aff53b9c84/konnectivity-agent/0.log"
Sep 16 19:29:44.501271 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:44.501236    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-29.ec2.internal_fdf7e40390d26689763c696c18fb8f21/haproxy/0.log"
Sep 16 19:29:49.149467 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:49.149437    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n6l4w_456725a2-50b4-402d-b137-062def6de1f8/node-exporter/0.log"
Sep 16 19:29:49.182026 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:49.181999    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n6l4w_456725a2-50b4-402d-b137-062def6de1f8/kube-rbac-proxy/0.log"
Sep 16 19:29:49.223483 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:49.223460    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n6l4w_456725a2-50b4-402d-b137-062def6de1f8/init-textfile/0.log"
Sep 16 19:29:52.070573 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070536    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"]
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070824    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070838    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070853    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kube-rbac-proxy"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070858    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kube-rbac-proxy"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070865    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070871    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070890    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070897    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070904    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kube-rbac-proxy"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070909    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kube-rbac-proxy"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070917    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kube-rbac-proxy"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070922    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kube-rbac-proxy"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070966    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kserve-container"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070973    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb15c3d-6a36-4c0e-8b85-9a19c54be6e9" containerName="kube-rbac-proxy"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070978    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kube-rbac-proxy"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070985    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8205d141-2120-49b0-bf31-f5c5c1156e5d" containerName="kserve-container"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070991    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kserve-container"
Sep 16 19:29:52.071011 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.070996    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9604a92a-ce44-4c0c-b3d4-4c00fa89281b" containerName="kube-rbac-proxy"
Sep 16 19:29:52.073679 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.073663    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.076696 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.076671    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sc8ck\"/\"default-dockercfg-zs942\""
Sep 16 19:29:52.077938 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.077919    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sc8ck\"/\"openshift-service-ca.crt\""
Sep 16 19:29:52.078044 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.077952    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sc8ck\"/\"kube-root-ca.crt\""
Sep 16 19:29:52.087649 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.087612    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"]
Sep 16 19:29:52.194934 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.194895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-lib-modules\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.195115 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.194947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24qn\" (UniqueName: \"kubernetes.io/projected/21063993-c1da-413c-8bdf-f48e2f6a50e3-kube-api-access-f24qn\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.195115 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.194982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-sys\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.195115 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.195019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-proc\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.195115 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.195053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-podres\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.296239 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.296200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24qn\" (UniqueName: \"kubernetes.io/projected/21063993-c1da-413c-8bdf-f48e2f6a50e3-kube-api-access-f24qn\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.296376 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.296249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-sys\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.296376 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.296275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-proc\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.296376 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.296315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-podres\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.296376 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.296338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-lib-modules\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.296553 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.296379    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-sys\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.296553 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.296432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-proc\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.296553 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.296454    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-lib-modules\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.296553 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.296478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21063993-c1da-413c-8bdf-f48e2f6a50e3-podres\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.310736 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.310708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24qn\" (UniqueName: \"kubernetes.io/projected/21063993-c1da-413c-8bdf-f48e2f6a50e3-kube-api-access-f24qn\") pod \"perf-node-gather-daemonset-25s8r\" (UID: \"21063993-c1da-413c-8bdf-f48e2f6a50e3\") " pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.383192 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.383130    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:52.661959 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:52.661875    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"]
Sep 16 19:29:52.664767 ip-10-0-128-29 kubenswrapper[2561]: W0916 19:29:52.664733    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21063993_c1da_413c_8bdf_f48e2f6a50e3.slice/crio-4c4d7cd4da3f05a32669160e0ba043bd74ada64d1bd13fa1d9a73774a48b2040 WatchSource:0}: Error finding container 4c4d7cd4da3f05a32669160e0ba043bd74ada64d1bd13fa1d9a73774a48b2040: Status 404 returned error can't find the container with id 4c4d7cd4da3f05a32669160e0ba043bd74ada64d1bd13fa1d9a73774a48b2040
Sep 16 19:29:53.601489 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:53.601448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r" event={"ID":"21063993-c1da-413c-8bdf-f48e2f6a50e3","Type":"ContainerStarted","Data":"45d9a498cbdb4c732db1667920714db6d2d316f522c46c16d3c772c71ce95deb"}
Sep 16 19:29:53.601874 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:53.601503    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r" event={"ID":"21063993-c1da-413c-8bdf-f48e2f6a50e3","Type":"ContainerStarted","Data":"4c4d7cd4da3f05a32669160e0ba043bd74ada64d1bd13fa1d9a73774a48b2040"}
Sep 16 19:29:53.601874 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:53.601555    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:29:53.624857 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:53.624802    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r" podStartSLOduration=1.62478612 podStartE2EDuration="1.62478612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:53.623740108 +0000 UTC m=+2964.978604745" watchObservedRunningTime="2026-09-16 19:29:53.62478612 +0000 UTC m=+2964.979650757"
Sep 16 19:29:54.491985 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:54.491948    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nnnm2_2fca5262-86ac-490a-b880-b2f620c6b019/dns/0.log"
Sep 16 19:29:54.521355 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:54.521326    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nnnm2_2fca5262-86ac-490a-b880-b2f620c6b019/kube-rbac-proxy/0.log"
Sep 16 19:29:54.548201 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:54.548179    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l8qxv_73d882ea-dbac-48fe-9406-ad43a64e7170/dns-node-resolver/0.log"
Sep 16 19:29:55.088465 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:55.088432    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86ddc94b5d-ktgfr_49cf8e26-f602-4ae6-84c3-e979a00ee6a9/registry/0.log"
Sep 16 19:29:55.179600 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:55.179573    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vxjwf_b84db093-de67-44ec-ade5-7a4a84648e0d/node-ca/0.log"
Sep 16 19:29:56.425680 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:56.425657    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mhrgc_469beb06-0eab-4e87-8a9d-23da328d232e/serve-healthcheck-canary/0.log"
Sep 16 19:29:57.077863 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:57.077834    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pxv54_817be5e4-45cb-4cb7-9aff-5f7b674aedc4/kube-rbac-proxy/0.log"
Sep 16 19:29:57.105153 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:57.105109    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pxv54_817be5e4-45cb-4cb7-9aff-5f7b674aedc4/exporter/0.log"
Sep 16 19:29:57.132606 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:57.132576    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pxv54_817be5e4-45cb-4cb7-9aff-5f7b674aedc4/extractor/0.log"
Sep 16 19:29:59.613800 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:29:59.613775    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sc8ck/perf-node-gather-daemonset-25s8r"
Sep 16 19:30:04.675630 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:04.675592    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-8skc5_69e62a94-7d95-4502-a51e-79c01704a516/migrator/0.log"
Sep 16 19:30:04.702182 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:04.702156    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-8skc5_69e62a94-7d95-4502-a51e-79c01704a516/graceful-termination/0.log"
Sep 16 19:30:06.681875 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:06.681845    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pp8w7_e2a384ac-d60b-4bbd-872d-dc42760a3455/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:30:06.707572 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:06.707538    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pp8w7_e2a384ac-d60b-4bbd-872d-dc42760a3455/egress-router-binary-copy/0.log"
Sep 16 19:30:06.732598 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:06.732573    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pp8w7_e2a384ac-d60b-4bbd-872d-dc42760a3455/cni-plugins/0.log"
Sep 16 19:30:06.755876 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:06.755853    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pp8w7_e2a384ac-d60b-4bbd-872d-dc42760a3455/bond-cni-plugin/0.log"
Sep 16 19:30:06.783429 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:06.783403    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pp8w7_e2a384ac-d60b-4bbd-872d-dc42760a3455/routeoverride-cni/0.log"
Sep 16 19:30:06.810774 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:06.810732    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pp8w7_e2a384ac-d60b-4bbd-872d-dc42760a3455/whereabouts-cni-bincopy/0.log"
Sep 16 19:30:06.840139 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:06.840113    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pp8w7_e2a384ac-d60b-4bbd-872d-dc42760a3455/whereabouts-cni/0.log"
Sep 16 19:30:06.879477 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:06.879451    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gf5lr_43c13f0d-db2b-46e7-87b6-50fa4666087a/kube-multus/0.log"
Sep 16 19:30:07.041948 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:07.041915    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dwzxx_80becee4-0e8e-437c-955b-e9cb44506e22/network-metrics-daemon/0.log"
Sep 16 19:30:07.073359 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:07.073329    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dwzxx_80becee4-0e8e-437c-955b-e9cb44506e22/kube-rbac-proxy/0.log"
Sep 16 19:30:08.414827 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:08.414785    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7clt_3dd8fb7f-cee2-4625-af27-51acec6ecdc6/ovn-controller/0.log"
Sep 16 19:30:08.437853 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:08.437814    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7clt_3dd8fb7f-cee2-4625-af27-51acec6ecdc6/ovn-acl-logging/0.log"
Sep 16 19:30:08.465362 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:08.465322    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7clt_3dd8fb7f-cee2-4625-af27-51acec6ecdc6/kube-rbac-proxy-node/0.log"
Sep 16 19:30:08.490553 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:08.490528    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7clt_3dd8fb7f-cee2-4625-af27-51acec6ecdc6/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:30:08.517335 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:08.517281    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7clt_3dd8fb7f-cee2-4625-af27-51acec6ecdc6/northd/0.log"
Sep 16 19:30:08.542231 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:08.542201    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7clt_3dd8fb7f-cee2-4625-af27-51acec6ecdc6/nbdb/0.log"
Sep 16 19:30:08.575695 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:08.575679    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7clt_3dd8fb7f-cee2-4625-af27-51acec6ecdc6/sbdb/0.log"
Sep 16 19:30:08.636049 ip-10-0-128-29 kubenswrapper[2561]: I0916 19:30:08.636023    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z7clt_3dd8fb7f-cee2-4625-af27-51acec6ecdc6/ovnkube-controller/0.log"