Jul 28 10:05:25.988697 ip-10-0-135-126 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 10:05:25.988709 ip-10-0-135-126 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 10:05:25.988717 ip-10-0-135-126 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 10:05:25.989027 ip-10-0-135-126 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 10:05:36.224349 ip-10-0-135-126 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 10:05:36.224367 ip-10-0-135-126 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a138d137f0a14504874e23d72e9dbd4e -- Jul 28 10:07:35.446916 ip-10-0-135-126 systemd[1]: Starting Kubernetes Kubelet... Jul 28 10:07:35.818979 ip-10-0-135-126 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 10:07:35.818979 ip-10-0-135-126 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 10:07:35.818979 ip-10-0-135-126 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 10:07:35.818979 ip-10-0-135-126 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 10:07:35.818979 ip-10-0-135-126 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 10:07:35.820482 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.820396 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 10:07:35.826120 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826105 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:07:35.826120 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826121 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826127 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826131 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826134 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826137 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826140 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826143 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826146 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826149 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826152 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826155 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826164 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826167 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826170 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826172 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826175 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826178 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826180 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826183 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:07:35.826187 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826185 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826188 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826191 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826193 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826196 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826199 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826201 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826204 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826206 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826209 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826211 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826214 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826217 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826220 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826222 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826225 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826228 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826231 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826234 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826236 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:07:35.826729 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826239 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826241 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826243 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826246 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826248 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826251 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826253 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826256 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826258 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826261 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826263 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826266 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826268 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826270 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826274 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826277 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826280 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826282 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826285 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826287 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:07:35.827489 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826290 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826292 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826295 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826297 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826300 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826302 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826305 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826307 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826310 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826312 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826315 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826317 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826320 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826323 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826325 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826328 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826331 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826333 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826336 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826339 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:07:35.828041 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826341 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826343 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826346 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826349 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826351 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826354 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826357 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826360 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826363 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826366 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826368 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826371 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826374 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826376 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826379 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826381 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826384 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826387 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:07:35.828522 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.826391 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828717 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828728 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828731 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828735 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828739 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828741 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828744 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828746 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828749 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828751 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828754 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828757 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828759 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828762 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828764 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828767 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828769 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828774 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828778 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:07:35.829066 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828780 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828783 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828785 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828788 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828791 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828793 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828796 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828798 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828801 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828803 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828806 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828808 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828810 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828814 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828817 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828820 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828823 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828826 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828829 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828831 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:07:35.829576 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828834 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828837 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828839 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828843 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828846 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828848 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828850 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828853 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828857 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828860 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828863 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828866 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828869 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828871 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828874 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828876 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828879 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828881 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828884 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828886 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:07:35.830086 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828888 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828891 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828893 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828895 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828898 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828901 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828904 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828907 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828909 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828912 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828915 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828917 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828920 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828923 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828925 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828928 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828930 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828933 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828936 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828939 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:07:35.830565 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828942 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828944 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828947 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828949 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828953 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828955 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828958 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828960 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828963 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828966 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828968 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828971 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828974 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828976 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828979 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828981 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828985 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828989 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828991 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:07:35.831059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.828994 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829362 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829366 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829369 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829372 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829374 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829377 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829380 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829382 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829385 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829387 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829390 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829392 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829395 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829398 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829400 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829403 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829406 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829409 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829412 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:07:35.831508 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829414 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829417 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829420 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829423 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829425 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829429 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829433 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829436 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829439 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829441 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829444 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829446 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829449 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829451 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829454 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829456 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829459 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829461 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829463 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829466 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:07:35.832000 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829468 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829470 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829473 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829475 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829478 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829481 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829484 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829487 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829489 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829492 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829495 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829497 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829499 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829502 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829504 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829506 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829509 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829512 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829514 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:07:35.832484 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829516 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829519 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829522 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829524 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829527 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829529 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829532 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829534 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829536 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829540 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829542 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829545 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829547 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829550 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829552 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829555 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829557 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829559 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829564 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829567 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:07:35.833003 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829570 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829572 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829575 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829578 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829581 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829584 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829587 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829590 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829592 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829595 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829597 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829600 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829602 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829606 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829610 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829613 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829616 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829636 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829639 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:07:35.833488 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829643 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.829647 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829723 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829732 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829745 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829749 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829753 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829756 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829760 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829765 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829768 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829773 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829776 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829779 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829782 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829785 2569 flags.go:64] FLAG: --cgroup-root="" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829788 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829791 2569 flags.go:64] FLAG: --client-ca-file="" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829795 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829798 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829803 2569 flags.go:64] FLAG: --cluster-domain="" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829806 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829809 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829812 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 10:07:35.833942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829815 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829819 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829822 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829825 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829829 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829832 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829835 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829838 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829840 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829843 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829847 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829850 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829853 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829856 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829859 2569 flags.go:64] FLAG: --enable-server="true" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829862 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829866 2569 flags.go:64] FLAG: --event-burst="100" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829869 2569 flags.go:64] FLAG: --event-qps="50" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829872 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829877 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829880 2569 flags.go:64] FLAG: --eviction-hard="" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829884 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829886 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829889 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829892 2569 flags.go:64] FLAG: --eviction-soft="" Jul 28 10:07:35.834501 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829895 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829898 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829901 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829904 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829907 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829910 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829913 2569 flags.go:64] FLAG: --feature-gates="" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829916 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829919 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829923 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829926 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829929 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829932 2569 flags.go:64] FLAG: --help="false" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829935 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-135-126.ec2.internal" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829938 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829941 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829944 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829947 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829950 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829953 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829956 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829959 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829962 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 10:07:35.835108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829965 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829968 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829971 2569 flags.go:64] FLAG: --kube-reserved="" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829975 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829978 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829981 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829984 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829986 2569 flags.go:64] FLAG: --lock-file="" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829989 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829993 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.829996 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830005 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830009 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830011 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830015 2569 flags.go:64] FLAG: --logging-format="text" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830018 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830021 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830024 2569 flags.go:64] FLAG: --manifest-url="" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830028 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830032 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830035 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830041 2569 flags.go:64] FLAG: --max-pods="110" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830044 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830047 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830050 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 10:07:35.835700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830053 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830055 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830058 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830061 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830069 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830072 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830075 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830078 2569 flags.go:64] FLAG: --pod-cidr="" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830081 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830087 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830091 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830095 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830098 2569 flags.go:64] FLAG: --port="10250" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830101 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830104 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d27afed6a7abd711" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830107 2569 flags.go:64] FLAG: --qos-reserved="" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830110 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830113 2569 flags.go:64] FLAG: --register-node="true" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830116 2569 flags.go:64] FLAG: --register-with-taints="" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830120 2569 flags.go:64] FLAG: --registry-burst="10" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830123 2569 flags.go:64] FLAG: --registry-qps="5" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830126 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830129 2569 flags.go:64] FLAG: --reserved-memory="" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830132 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830135 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 10:07:35.836346 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830138 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830142 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830145 2569 flags.go:64] FLAG: --runonce="false" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830148 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830151 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830154 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830156 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830159 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830162 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830165 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830167 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830170 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830173 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830176 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830179 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830181 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830184 2569 flags.go:64] FLAG: --system-cgroups="" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830189 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830194 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830196 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830199 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830204 2569 flags.go:64] FLAG: --tls-min-version="" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830207 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830210 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830213 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 10:07:35.837045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830216 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830219 2569 flags.go:64] FLAG: --v="2" Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830223 2569 flags.go:64] FLAG: --version="false" Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830227 2569 flags.go:64] FLAG: --vmodule="" Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830231 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.830234 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830348 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830352 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830357 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830361 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830364 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830366 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830369 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830371 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830374 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830377 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830379 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830382 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830384 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830387 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:07:35.837680 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830389 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830392 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830396 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830399 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830403 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830406 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830408 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830411 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830414 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830416 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830419 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830421 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830424 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830426 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830429 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830431 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830435 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830438 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830440 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830442 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:07:35.838183 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830446 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830449 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830451 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830454 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830457 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830459 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830462 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830464 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830467 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830469 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830472 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830475 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830477 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830480 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830483 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830485 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830489 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830492 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830494 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830496 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:07:35.838691 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830499 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830501 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830504 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830506 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830509 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830511 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830514 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830516 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830519 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830521 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830524 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830526 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830530 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830533 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830535 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830538 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830540 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830543 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830545 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830548 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:07:35.839177 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830550 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830553 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830555 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830558 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830561 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830563 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830566 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830569 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830573 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830575 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830578 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830582 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830585 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830588 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830591 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830594 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830597 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830599 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830602 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:07:35.839670 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830604 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830607 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830610 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830613 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830615 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.830631 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.831744 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.838555 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.838568 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.838587 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.838593 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838664 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838670 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838674 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838677 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:07:35.840139 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838679 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838682 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838685 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838688 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838692 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838695 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838699 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838702 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838705 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838707 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838710 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838713 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838715 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838718 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838721 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838723 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838726 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838728 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838731 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838734 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:07:35.840512 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838737 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838739 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838742 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838745 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838748 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838751 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838753 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838756 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838759 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838761 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838764 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838766 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838769 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838772 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838774 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838777 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838780 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838782 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838785 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838788 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:07:35.841064 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838790 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838793 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838795 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838798 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838800 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838803 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838805 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838808 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838810 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838813 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838815 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838818 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838820 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838824 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838827 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838831 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838835 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838838 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838841 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:07:35.841546 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838845 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838848 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838850 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838853 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838856 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838858 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838861 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838863 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838866 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838868 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838871 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838873 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838875 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838879 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838882 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838885 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838887 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838890 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838893 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838896 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:07:35.842059 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838898 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838901 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838903 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838906 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838909 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838912 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838914 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838917 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838920 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838923 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838926 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838928 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838931 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838935 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838939 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:07:35.842538 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.838941 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.838947 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839059 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839065 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839068 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839071 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839073 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839076 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839078 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839081 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839084 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839087 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839089 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839092 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 10:07:35.842918 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839094 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839097 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839099 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839102 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839105 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839107 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839110 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839112 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839115 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839118 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839121 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839124 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839127 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839129 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839132 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839134 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839137 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839139 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839142 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839144 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839146 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 10:07:35.843260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839149 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839151 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839154 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839156 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839159 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839161 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839163 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839166 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839169 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839172 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839175 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839177 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839180 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839183 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839186 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839189 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839191 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839194 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839197 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 10:07:35.843833 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839200 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839202 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839205 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839207 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839211 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839213 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839215 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839218 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839220 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839223 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839226 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839228 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839231 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839233 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839236 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839238 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839241 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839243 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839245 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839248 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 10:07:35.844310 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839250 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839253 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839255 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839258 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839260 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839263 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839267 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839270 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839273 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839276 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839279 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839281 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839284 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839286 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839289 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839291 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839294 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839296 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839299 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 10:07:35.844924 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839302 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839304 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839306 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839309 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839311 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839313 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839316 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:35.839319 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.839323 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.839427 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.843289 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.844528 2569 server.go:1019] "Starting client certificate rotation" Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.844659 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 10:07:35.845420 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.844715 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 10:07:35.846771 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.846756 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 10:07:35.850258 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.850238 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 10:07:35.859117 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.859099 2569 log.go:25] "Validated CRI v1 runtime API" Jul 28 10:07:35.862204 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.862186 2569 log.go:25] "Validated CRI v1 image API" Jul 28 10:07:35.863021 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.863006 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 10:07:35.869286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.869266 2569 fs.go:135] Filesystem UUIDs: map[0a19cad5-fed6-4530-af1e-e8004abf0b78:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a0d1d336-1ac4-444b-a424-6ee1657597b2:/dev/nvme0n1p4] Jul 28 10:07:35.869338 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.869286 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 10:07:35.874459 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.874360 2569 manager.go:216] Machine: {Timestamp:2026-07-28 10:07:35.873277864 +0000 UTC m=+0.315856829 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091545 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2b3b06f834b688d2334693ea49f3 SystemUUID:ec2b2b3b-06f8-34b6-88d2-334693ea49f3 BootID:a138d137-f0a1-4504-874e-23d72e9dbd4e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bf:0c:0b:ea:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bf:0c:0b:ea:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:01:41:0c:41:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 10:07:35.874459 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.874454 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 10:07:35.874572 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.874561 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 10:07:35.875764 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.875741 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 10:07:35.878057 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.878035 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 10:07:35.878194 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.878061 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-126.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 10:07:35.878781 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.878771 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 10:07:35.878816 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.878782 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 10:07:35.878816 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.878794 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 10:07:35.880323 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.880313 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 10:07:35.881571 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.881561 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 10:07:35.884557 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.884545 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 28 10:07:35.884694 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.884685 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 10:07:35.884724 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.884713 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 28 10:07:35.884724 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.884723 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 10:07:35.884774 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.884736 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 10:07:35.884774 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.884746 2569 kubelet.go:398] "Adding apiserver pod source" Jul 28 10:07:35.884774 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.884754 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 10:07:35.885098 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.885087 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 10:07:35.885134 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.885110 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 10:07:35.887136 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.887118 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 10:07:35.888402 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.888388 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 10:07:35.888450 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.888420 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 10:07:35.889402 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889391 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 10:07:35.889436 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889406 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 10:07:35.889436 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889412 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 10:07:35.889436 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889417 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 10:07:35.889436 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889426 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 10:07:35.889436 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889431 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 10:07:35.889560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889439 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 10:07:35.889560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889444 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 10:07:35.889560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889450 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 10:07:35.889560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889455 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 10:07:35.889560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889465 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 10:07:35.889560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.889472 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 10:07:35.890879 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.890868 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 10:07:35.890920 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.890881 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 10:07:35.895397 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.895253 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 10:07:35.895397 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.895329 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 10:07:35.895532 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.895470 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-126.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 10:07:35.895849 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.895733 2569 server.go:1268] "Started kubelet" Jul 28 10:07:35.895962 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.895912 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 10:07:35.896028 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.895986 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 10:07:35.896118 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.896089 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 10:07:35.896810 ip-10-0-135-126 systemd[1]: Started Kubernetes Kubelet. Jul 28 10:07:35.897558 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.897424 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 10:07:35.898350 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.898337 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 28 10:07:35.900673 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.900643 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 10:07:35.904370 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.903488 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-126.ec2.internal.18c66c300ae08d0c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-126.ec2.internal,UID:ip-10-0-135-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-126.ec2.internal,},FirstTimestamp:2026-07-28 10:07:35.8956987 +0000 UTC m=+0.338277671,LastTimestamp:2026-07-28 10:07:35.8956987 +0000 UTC m=+0.338277671,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-126.ec2.internal,}" Jul 28 10:07:35.905562 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.905546 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 10:07:35.905562 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.905550 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 10:07:35.905712 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.905609 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:35.906102 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.906084 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 10:07:35.907290 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.906085 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nwml4" Jul 28 10:07:35.907290 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.906094 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 10:07:35.907290 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.906898 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 10:07:35.907290 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.907005 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 28 10:07:35.907290 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.907013 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 28 10:07:35.907943 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.907926 2569 factory.go:55] Registering systemd factory Jul 28 10:07:35.907943 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.907945 2569 factory.go:223] Registration of the systemd container factory successfully Jul 28 10:07:35.908310 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.908293 2569 factory.go:153] Registering CRI-O factory Jul 28 10:07:35.908373 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.908314 2569 factory.go:223] Registration of the crio container factory successfully Jul 28 10:07:35.908373 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.908365 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 10:07:35.908437 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.908388 2569 factory.go:103] Registering Raw factory Jul 28 10:07:35.908437 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.908403 2569 manager.go:1196] Started watching for new ooms in manager Jul 28 10:07:35.908859 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.908847 2569 manager.go:349] Starting recovery of all containers Jul 28 10:07:35.914271 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.914248 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nwml4" Jul 28 10:07:35.915150 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.915124 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 10:07:35.915150 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.915124 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 10:07:35.919532 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.919520 2569 manager.go:354] Recovery completed Jul 28 10:07:35.924211 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.924197 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:07:35.926878 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.926828 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:07:35.926878 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.926862 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:07:35.926878 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.926878 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:07:35.927773 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.927753 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 10:07:35.927773 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.927768 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:07:35.927908 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.927772 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 10:07:35.927908 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.927829 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 28 10:07:35.928690 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.928659 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:07:35.928780 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.928700 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:07:35.928780 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.928717 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:07:35.928933 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.928918 2569 policy_none.go:49] "None policy: Start" Jul 28 10:07:35.928988 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.928941 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 10:07:35.928988 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.928956 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 10:07:35.930121 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.930106 2569 policy_none.go:47] "Start" Jul 28 10:07:35.969274 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.969151 2569 manager.go:337] "Starting Device Plugin manager" Jul 28 10:07:35.969356 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.969297 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 10:07:35.969356 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.969311 2569 server.go:87] "Starting device plugin registration server" Jul 28 10:07:35.969530 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.969519 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 10:07:35.969563 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.969532 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 10:07:35.969658 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.969642 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 10:07:35.969813 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.969800 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 10:07:35.969813 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:35.969814 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 10:07:35.970256 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.970236 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 10:07:35.970310 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:35.970284 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.069735 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.069666 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:07:36.070812 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.070595 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:07:36.070812 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.070637 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:07:36.070812 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.070648 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:07:36.070812 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.070675 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.071414 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.071393 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 10:07:36.072792 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.072767 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 10:07:36.072869 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.072813 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 10:07:36.072869 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.072837 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 10:07:36.072947 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.072871 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 10:07:36.074557 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.074538 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:07:36.083300 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.083284 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.083379 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.083304 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-126.ec2.internal\": node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.118086 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.118063 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.173562 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.173531 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal"] Jul 28 10:07:36.173679 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.173619 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:07:36.174916 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.174897 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:07:36.174978 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.174932 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:07:36.174978 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.174943 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:07:36.176448 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.176432 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:07:36.176587 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.176570 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.176628 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.176606 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:07:36.177291 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.177274 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:07:36.177373 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.177306 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:07:36.177373 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.177282 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:07:36.177373 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.177338 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:07:36.177373 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.177349 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:07:36.177373 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.177317 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:07:36.178712 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.178686 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.178767 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.178731 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 10:07:36.179486 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.179466 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientMemory" Jul 28 10:07:36.179486 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.179488 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 10:07:36.179609 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.179499 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeHasSufficientPID" Jul 28 10:07:36.203350 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.203324 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-126.ec2.internal\" not found" node="ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.207861 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.207844 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-126.ec2.internal\" not found" node="ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.208987 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.208966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33e04923af854eb13d5fa434fe913350-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal\" (UID: \"33e04923af854eb13d5fa434fe913350\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.209039 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.208996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33e04923af854eb13d5fa434fe913350-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal\" (UID: \"33e04923af854eb13d5fa434fe913350\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.209039 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.209013 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05b32d16ed33041d1fceff9e2d07ef80-config\") pod \"kube-apiserver-proxy-ip-10-0-135-126.ec2.internal\" (UID: \"05b32d16ed33041d1fceff9e2d07ef80\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.218756 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.218738 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.309286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.309260 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33e04923af854eb13d5fa434fe913350-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal\" (UID: \"33e04923af854eb13d5fa434fe913350\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.309371 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.309293 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33e04923af854eb13d5fa434fe913350-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal\" (UID: \"33e04923af854eb13d5fa434fe913350\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.309371 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.309300 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05b32d16ed33041d1fceff9e2d07ef80-config\") pod \"kube-apiserver-proxy-ip-10-0-135-126.ec2.internal\" (UID: \"05b32d16ed33041d1fceff9e2d07ef80\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.309371 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.309335 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05b32d16ed33041d1fceff9e2d07ef80-config\") pod \"kube-apiserver-proxy-ip-10-0-135-126.ec2.internal\" (UID: \"05b32d16ed33041d1fceff9e2d07ef80\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.309371 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.309352 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33e04923af854eb13d5fa434fe913350-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal\" (UID: \"33e04923af854eb13d5fa434fe913350\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.309521 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.309388 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33e04923af854eb13d5fa434fe913350-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal\" (UID: \"33e04923af854eb13d5fa434fe913350\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.319467 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.319446 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.419889 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.419802 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.512120 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.512094 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.512199 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.512166 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.519899 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.519880 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.620520 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.620476 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.721215 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.721135 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.733593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.733568 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:07:36.821363 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.821331 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-126.ec2.internal\" not found" Jul 28 10:07:36.834641 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.834615 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:07:36.843872 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.843848 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 10:07:36.844008 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.843986 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 10:07:36.844058 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.844012 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 10:07:36.844058 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.844026 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 10:07:36.887013 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.886981 2569 apiserver.go:52] "Watching apiserver" Jul 28 10:07:36.906212 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.906155 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 10:07:36.906369 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.906157 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.907484 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.907466 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 10:07:36.907848 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.907828 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-qdg7w","kube-system/global-pull-secret-syncer-q6r8g","kube-system/konnectivity-agent-r4n9j","openshift-cluster-node-tuning-operator/tuned-b2j8p","openshift-image-registry/node-ca-mj48w","openshift-multus/multus-z899d","openshift-multus/network-metrics-daemon-trrp4","openshift-network-diagnostics/network-check-target-fr92k","openshift-ovn-kubernetes/ovnkube-node-wkd29","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c","openshift-dns/node-resolver-tp929","openshift-multus/multus-additional-cni-plugins-75jhn"] Jul 28 10:07:36.909505 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.909489 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:36.910661 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.910640 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:36.910762 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.910716 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:36.911962 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.911944 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:07:36.912601 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.912575 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:07:36.912753 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.912619 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-prmcj\"" Jul 28 10:07:36.912883 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.912869 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 10:07:36.913743 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.913727 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:36.914144 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.914126 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 10:07:36.914913 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.914896 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g9hk5\"" Jul 28 10:07:36.914986 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.914947 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 10:07:36.915414 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.915395 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:36.915477 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.915461 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 10:07:36.916628 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.916613 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mnznw\"" Jul 28 10:07:36.916860 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.916846 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 10:07:36.917294 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.917273 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:07:36.917733 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.917698 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 10:02:36 +0000 UTC" deadline="2028-03-17 17:15:17.117647789 +0000 UTC" Jul 28 10:07:36.917733 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.917731 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14359h7m40.199919007s" Jul 28 10:07:36.917881 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.917799 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z899d" Jul 28 10:07:36.919606 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.919590 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:36.919697 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.919659 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:36.920225 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.920209 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 10:07:36.920298 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.920264 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 10:07:36.920474 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.920459 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hl8q4\"" Jul 28 10:07:36.920521 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.920459 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 10:07:36.921075 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.921060 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:36.921154 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:36.921119 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:36.921863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.921844 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 10:07:36.921983 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.921964 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal" Jul 28 10:07:36.922336 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.922314 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 10:07:36.923029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.922596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:36.923321 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.923263 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 10:07:36.924088 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.923880 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 10:07:36.924088 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.923979 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w7679\"" Jul 28 10:07:36.924088 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.924061 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 10:07:36.924281 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.924249 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:36.925457 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.925435 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 10:07:36.926281 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.926257 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 10:07:36.926578 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.926560 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:36.926734 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.926592 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vrs9p\"" Jul 28 10:07:36.926734 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.926714 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 10:07:36.926734 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.926724 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 10:07:36.926918 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.926777 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 10:07:36.926918 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.926807 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 10:07:36.927759 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.927744 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 10:07:36.928002 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.927987 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 10:07:36.928186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.928170 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pwk2p\"" Jul 28 10:07:36.928244 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.928170 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 10:07:36.928244 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.928201 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 10:07:36.928416 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.928398 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal"] Jul 28 10:07:36.928514 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.928501 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:36.929445 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.929429 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-psq46\"" Jul 28 10:07:36.929850 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.929835 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 10:07:36.933601 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.933583 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal"] Jul 28 10:07:36.934068 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.934054 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 10:07:36.934215 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.934200 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 10:07:36.934434 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.934422 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 10:07:36.934872 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.934857 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 10:07:36.934924 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.934903 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8m7kv\"" Jul 28 10:07:36.959198 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.959179 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k629l" Jul 28 10:07:36.972945 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:36.972884 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k629l" Jul 28 10:07:37.008024 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.008005 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 10:07:37.012745 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-run\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.012823 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-cni-bin\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.012823 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012777 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-sys-fs\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.012823 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012804 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e564183a-fced-4212-9fcf-6c1ffbf65cfa-konnectivity-ca\") pod \"konnectivity-agent-r4n9j\" (UID: \"e564183a-fced-4212-9fcf-6c1ffbf65cfa\") " pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmwf\" (UniqueName: \"kubernetes.io/projected/1acd164c-032f-40eb-bef6-b07042848a90-kube-api-access-wfmwf\") pod \"node-ca-mj48w\" (UID: \"1acd164c-032f-40eb-bef6-b07042848a90\") " pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012839 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022e6ace-b742-4eb3-a13a-97d1ec1f10e3-tmp-dir\") pod \"node-resolver-tp929\" (UID: \"022e6ace-b742-4eb3-a13a-97d1ec1f10e3\") " pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012854 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-sysctl-conf\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012881 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-systemd-units\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-socket-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-etc-selinux\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzks7\" (UniqueName: \"kubernetes.io/projected/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-kube-api-access-mzks7\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012956 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvdlx\" (UniqueName: \"kubernetes.io/projected/333ddf02-969b-40cf-8d27-bb408a5a3873-kube-api-access-vvdlx\") pod \"iptables-alerter-qdg7w\" (UID: \"333ddf02-969b-40cf-8d27-bb408a5a3873\") " pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.012973 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-var-lib-cni-multus\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.012993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12f63461-f09b-4642-b556-023c9f371845-ovn-node-metrics-cert\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013031 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-system-cni-dir\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbac8868-5bb5-4857-895f-455556932955-dbus\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013064 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-run-netns\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013080 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-kubelet\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013103 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kfw\" (UniqueName: \"kubernetes.io/projected/022e6ace-b742-4eb3-a13a-97d1ec1f10e3-kube-api-access-k2kfw\") pod \"node-resolver-tp929\" (UID: \"022e6ace-b742-4eb3-a13a-97d1ec1f10e3\") " pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-lib-modules\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7x7j\" (UniqueName: \"kubernetes.io/projected/7062085c-ecbf-476b-b845-e594877f19d9-kube-api-access-n7x7j\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013178 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-slash\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013231 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-cnibin\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbac8868-5bb5-4857-895f-455556932955-kubelet-config\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013313 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013338 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1acd164c-032f-40eb-bef6-b07042848a90-serviceca\") pod \"node-ca-mj48w\" (UID: \"1acd164c-032f-40eb-bef6-b07042848a90\") " pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-multus-cni-dir\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013390 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-run-multus-certs\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-run-systemd\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.013483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013452 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-device-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013482 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013500 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-sysctl-d\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013533 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-multus-conf-dir\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013549 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-var-lib-openvswitch\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e564183a-fced-4212-9fcf-6c1ffbf65cfa-agent-certs\") pod \"konnectivity-agent-r4n9j\" (UID: \"e564183a-fced-4212-9fcf-6c1ffbf65cfa\") " pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013580 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcj26\" (UniqueName: \"kubernetes.io/projected/06b28783-48ec-4b28-94de-385cd15baaa8-kube-api-access-zcj26\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013596 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-sys\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013618 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-kubernetes\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-log-socket\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013672 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dea71310-9b24-40e8-8b58-8780a189b3df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013704 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-host\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-system-cni-dir\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013765 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv25m\" (UniqueName: \"kubernetes.io/projected/dea71310-9b24-40e8-8b58-8780a189b3df-kube-api-access-tv25m\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-tmp\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.014042 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013853 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-var-lib-kubelet\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-run-netns\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013903 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-run-openvswitch\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013925 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-run-ovn\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.013942 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-cni-netd\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014005 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/333ddf02-969b-40cf-8d27-bb408a5a3873-host-slash\") pod \"iptables-alerter-qdg7w\" (UID: \"333ddf02-969b-40cf-8d27-bb408a5a3873\") " pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-var-lib-kubelet\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014044 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccxx\" (UniqueName: \"kubernetes.io/projected/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-kube-api-access-rccxx\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014061 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-hostroot\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014075 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7062085c-ecbf-476b-b845-e594877f19d9-multus-daemon-config\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014089 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-etc-kubernetes\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-node-log\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014164 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12f63461-f09b-4642-b556-023c9f371845-env-overrides\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014192 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dea71310-9b24-40e8-8b58-8780a189b3df-cni-binary-copy\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014219 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/333ddf02-969b-40cf-8d27-bb408a5a3873-iptables-alerter-script\") pod \"iptables-alerter-qdg7w\" (UID: \"333ddf02-969b-40cf-8d27-bb408a5a3873\") " pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014234 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-tuned\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7062085c-ecbf-476b-b845-e594877f19d9-cni-binary-copy\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.014691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014288 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-multus-socket-dir-parent\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-etc-openvswitch\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014338 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-run-ovn-kubernetes\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014378 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014413 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014437 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-registration-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014463 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12f63461-f09b-4642-b556-023c9f371845-ovnkube-config\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014516 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-os-release\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014544 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dea71310-9b24-40e8-8b58-8780a189b3df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014571 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1acd164c-032f-40eb-bef6-b07042848a90-host\") pod \"node-ca-mj48w\" (UID: \"1acd164c-032f-40eb-bef6-b07042848a90\") " pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014596 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-cnibin\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014620 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-run-k8s-cni-cncf-io\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014717 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12f63461-f09b-4642-b556-023c9f371845-ovnkube-script-lib\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014745 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/022e6ace-b742-4eb3-a13a-97d1ec1f10e3-hosts-file\") pod \"node-resolver-tp929\" (UID: \"022e6ace-b742-4eb3-a13a-97d1ec1f10e3\") " pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014793 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-modprobe-d\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.015547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014817 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-sysconfig\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.016423 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-systemd\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.016423 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014882 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-os-release\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.016423 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014908 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-var-lib-cni-bin\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.016423 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.014944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sx4w\" (UniqueName: \"kubernetes.io/projected/12f63461-f09b-4642-b556-023c9f371845-kube-api-access-7sx4w\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115232 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115199 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-hostroot\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.115232 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7062085c-ecbf-476b-b845-e594877f19d9-multus-daemon-config\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.115435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-etc-kubernetes\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.115435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-node-log\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115330 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-node-log\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115340 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-hostroot\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.115435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12f63461-f09b-4642-b556-023c9f371845-env-overrides\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dea71310-9b24-40e8-8b58-8780a189b3df-cni-binary-copy\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.115435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115411 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/333ddf02-969b-40cf-8d27-bb408a5a3873-iptables-alerter-script\") pod \"iptables-alerter-qdg7w\" (UID: \"333ddf02-969b-40cf-8d27-bb408a5a3873\") " pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.115435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115426 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-tuned\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115340 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-etc-kubernetes\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7062085c-ecbf-476b-b845-e594877f19d9-cni-binary-copy\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115525 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-multus-socket-dir-parent\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-etc-openvswitch\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115588 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-run-ovn-kubernetes\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115616 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115661 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-registration-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12f63461-f09b-4642-b556-023c9f371845-ovnkube-config\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115735 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-multus-socket-dir-parent\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-os-release\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115792 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-etc-openvswitch\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115792 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dea71310-9b24-40e8-8b58-8780a189b3df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1acd164c-032f-40eb-bef6-b07042848a90-host\") pod \"node-ca-mj48w\" (UID: \"1acd164c-032f-40eb-bef6-b07042848a90\") " pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115839 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-run-ovn-kubernetes\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115850 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-cnibin\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.115863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115859 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12f63461-f09b-4642-b556-023c9f371845-env-overrides\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-run-k8s-cni-cncf-io\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12f63461-f09b-4642-b556-023c9f371845-ovnkube-script-lib\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115924 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7062085c-ecbf-476b-b845-e594877f19d9-multus-daemon-config\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115947 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-registration-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116000 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-cnibin\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116023 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-os-release\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116046 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116060 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7062085c-ecbf-476b-b845-e594877f19d9-cni-binary-copy\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116124 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/022e6ace-b742-4eb3-a13a-97d1ec1f10e3-hosts-file\") pod \"node-resolver-tp929\" (UID: \"022e6ace-b742-4eb3-a13a-97d1ec1f10e3\") " pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-modprobe-d\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-sysconfig\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116206 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-systemd\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116227 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/333ddf02-969b-40cf-8d27-bb408a5a3873-iptables-alerter-script\") pod \"iptables-alerter-qdg7w\" (UID: \"333ddf02-969b-40cf-8d27-bb408a5a3873\") " pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-os-release\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-var-lib-cni-bin\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116289 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-os-release\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.116839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116308 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-run-k8s-cni-cncf-io\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116343 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-var-lib-cni-bin\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116381 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-sysconfig\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116393 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-systemd\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sx4w\" (UniqueName: \"kubernetes.io/projected/12f63461-f09b-4642-b556-023c9f371845-kube-api-access-7sx4w\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116426 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/022e6ace-b742-4eb3-a13a-97d1ec1f10e3-hosts-file\") pod \"node-resolver-tp929\" (UID: \"022e6ace-b742-4eb3-a13a-97d1ec1f10e3\") " pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116451 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-run\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116475 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-cni-bin\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-sys-fs\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116506 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-modprobe-d\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e564183a-fced-4212-9fcf-6c1ffbf65cfa-konnectivity-ca\") pod \"konnectivity-agent-r4n9j\" (UID: \"e564183a-fced-4212-9fcf-6c1ffbf65cfa\") " pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116528 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dea71310-9b24-40e8-8b58-8780a189b3df-cni-binary-copy\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116544 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-run\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116497 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12f63461-f09b-4642-b556-023c9f371845-ovnkube-config\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116553 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmwf\" (UniqueName: \"kubernetes.io/projected/1acd164c-032f-40eb-bef6-b07042848a90-kube-api-access-wfmwf\") pod \"node-ca-mj48w\" (UID: \"1acd164c-032f-40eb-bef6-b07042848a90\") " pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116581 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-cni-bin\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116589 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dea71310-9b24-40e8-8b58-8780a189b3df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116602 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-sys-fs\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.117691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116621 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1acd164c-032f-40eb-bef6-b07042848a90-host\") pod \"node-ca-mj48w\" (UID: \"1acd164c-032f-40eb-bef6-b07042848a90\") " pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022e6ace-b742-4eb3-a13a-97d1ec1f10e3-tmp-dir\") pod \"node-resolver-tp929\" (UID: \"022e6ace-b742-4eb3-a13a-97d1ec1f10e3\") " pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.115889 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116767 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-sysctl-conf\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116791 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12f63461-f09b-4642-b556-023c9f371845-ovnkube-script-lib\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116809 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-systemd-units\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-socket-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116841 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116859 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-etc-selinux\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116882 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-systemd-units\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116888 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzks7\" (UniqueName: \"kubernetes.io/projected/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-kube-api-access-mzks7\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.116997 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvdlx\" (UniqueName: \"kubernetes.io/projected/333ddf02-969b-40cf-8d27-bb408a5a3873-kube-api-access-vvdlx\") pod \"iptables-alerter-qdg7w\" (UID: \"333ddf02-969b-40cf-8d27-bb408a5a3873\") " pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117000 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-socket-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117025 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-var-lib-cni-multus\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117043 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/022e6ace-b742-4eb3-a13a-97d1ec1f10e3-tmp-dir\") pod \"node-resolver-tp929\" (UID: \"022e6ace-b742-4eb3-a13a-97d1ec1f10e3\") " pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117055 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12f63461-f09b-4642-b556-023c9f371845-ovn-node-metrics-cert\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117082 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-system-cni-dir\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.118567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117107 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-var-lib-cni-multus\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117110 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbac8868-5bb5-4857-895f-455556932955-dbus\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117149 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-run-netns\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-kubelet\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kfw\" (UniqueName: \"kubernetes.io/projected/022e6ace-b742-4eb3-a13a-97d1ec1f10e3-kube-api-access-k2kfw\") pod \"node-resolver-tp929\" (UID: \"022e6ace-b742-4eb3-a13a-97d1ec1f10e3\") " pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-lib-modules\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117229 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbac8868-5bb5-4857-895f-455556932955-dbus\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117248 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7x7j\" (UniqueName: \"kubernetes.io/projected/7062085c-ecbf-476b-b845-e594877f19d9-kube-api-access-n7x7j\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-slash\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117283 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-system-cni-dir\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117303 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-cnibin\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117319 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e564183a-fced-4212-9fcf-6c1ffbf65cfa-konnectivity-ca\") pod \"konnectivity-agent-r4n9j\" (UID: \"e564183a-fced-4212-9fcf-6c1ffbf65cfa\") " pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbac8868-5bb5-4857-895f-455556932955-kubelet-config\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117369 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-slash\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117377 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-lib-modules\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117384 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1acd164c-032f-40eb-bef6-b07042848a90-serviceca\") pod \"node-ca-mj48w\" (UID: \"1acd164c-032f-40eb-bef6-b07042848a90\") " pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117409 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-multus-cni-dir\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.119428 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117417 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-cnibin\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-run-multus-certs\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.117460 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117463 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-run-systemd\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117502 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-sysctl-conf\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117511 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-multus-cni-dir\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117512 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-device-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117462 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbac8868-5bb5-4857-895f-455556932955-kubelet-config\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.117537 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret podName:cbac8868-5bb5-4857-895f-455556932955 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:37.617516253 +0000 UTC m=+2.060095230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret") pod "global-pull-secret-syncer-q6r8g" (UID: "cbac8868-5bb5-4857-895f-455556932955") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117566 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-device-dir\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117599 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-sysctl-d\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-multus-conf-dir\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117675 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-var-lib-openvswitch\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117684 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-run-systemd\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117683 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-kubelet\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117621 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-run-multus-certs\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.120404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e564183a-fced-4212-9fcf-6c1ffbf65cfa-agent-certs\") pod \"konnectivity-agent-r4n9j\" (UID: \"e564183a-fced-4212-9fcf-6c1ffbf65cfa\") " pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117767 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-multus-conf-dir\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcj26\" (UniqueName: \"kubernetes.io/projected/06b28783-48ec-4b28-94de-385cd15baaa8-kube-api-access-zcj26\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117797 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-sys\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117815 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-var-lib-openvswitch\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-kubernetes\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117816 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-run-netns\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-log-socket\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117772 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dea71310-9b24-40e8-8b58-8780a189b3df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117882 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-kubernetes\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117884 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dea71310-9b24-40e8-8b58-8780a189b3df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117917 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-host\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117942 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-system-cni-dir\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117968 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv25m\" (UniqueName: \"kubernetes.io/projected/dea71310-9b24-40e8-8b58-8780a189b3df-kube-api-access-tv25m\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.117994 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-tmp\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118037 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1acd164c-032f-40eb-bef6-b07042848a90-serviceca\") pod \"node-ca-mj48w\" (UID: \"1acd164c-032f-40eb-bef6-b07042848a90\") " pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118043 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-var-lib-kubelet\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.121314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118082 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-host\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118097 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-run-netns\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118124 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-run-openvswitch\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118139 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-sys\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118151 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-run-ovn\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118183 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-run-ovn\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-cni-netd\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118220 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-run-netns\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118229 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-log-socket\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118239 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-run-openvswitch\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118256 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12f63461-f09b-4642-b556-023c9f371845-host-cni-netd\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118263 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-host-var-lib-kubelet\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.118166 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/333ddf02-969b-40cf-8d27-bb408a5a3873-host-slash\") pod \"iptables-alerter-qdg7w\" (UID: \"333ddf02-969b-40cf-8d27-bb408a5a3873\") " pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118289 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dea71310-9b24-40e8-8b58-8780a189b3df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-var-lib-kubelet\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.118334 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs podName:06b28783-48ec-4b28-94de-385cd15baaa8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:37.618300486 +0000 UTC m=+2.060879456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs") pod "network-metrics-daemon-trrp4" (UID: "06b28783-48ec-4b28-94de-385cd15baaa8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:37.121934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118345 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-var-lib-kubelet\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.122567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118356 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7062085c-ecbf-476b-b845-e594877f19d9-system-cni-dir\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.122567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccxx\" (UniqueName: \"kubernetes.io/projected/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-kube-api-access-rccxx\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.122567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118397 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/333ddf02-969b-40cf-8d27-bb408a5a3873-host-slash\") pod \"iptables-alerter-qdg7w\" (UID: \"333ddf02-969b-40cf-8d27-bb408a5a3873\") " pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.122567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118401 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-etc-selinux\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.122567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.118517 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-sysctl-d\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.122567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.120604 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-etc-tuned\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.122567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.120618 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-tmp\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.122567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.120906 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12f63461-f09b-4642-b556-023c9f371845-ovn-node-metrics-cert\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.122567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.121063 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e564183a-fced-4212-9fcf-6c1ffbf65cfa-agent-certs\") pod \"konnectivity-agent-r4n9j\" (UID: \"e564183a-fced-4212-9fcf-6c1ffbf65cfa\") " pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:07:37.130902 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.130880 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sx4w\" (UniqueName: \"kubernetes.io/projected/12f63461-f09b-4642-b556-023c9f371845-kube-api-access-7sx4w\") pod \"ovnkube-node-wkd29\" (UID: \"12f63461-f09b-4642-b556-023c9f371845\") " pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.132643 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.132618 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:07:37.132783 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.132650 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:07:37.132783 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.132665 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4vdgc for pod openshift-network-diagnostics/network-check-target-fr92k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:37.132783 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.132738 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc podName:d6b7677d-6c6c-4be6-893c-936e9ca050be nodeName:}" failed. No retries permitted until 2026-07-28 10:07:37.632719943 +0000 UTC m=+2.075298922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4vdgc" (UniqueName: "kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc") pod "network-check-target-fr92k" (UID: "d6b7677d-6c6c-4be6-893c-936e9ca050be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:37.133609 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.133587 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccxx\" (UniqueName: \"kubernetes.io/projected/19bdd7fe-0327-4db4-83ad-38b5b334cbdc-kube-api-access-rccxx\") pod \"tuned-b2j8p\" (UID: \"19bdd7fe-0327-4db4-83ad-38b5b334cbdc\") " pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.133782 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.133726 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7x7j\" (UniqueName: \"kubernetes.io/projected/7062085c-ecbf-476b-b845-e594877f19d9-kube-api-access-n7x7j\") pod \"multus-z899d\" (UID: \"7062085c-ecbf-476b-b845-e594877f19d9\") " pod="openshift-multus/multus-z899d" Jul 28 10:07:37.135355 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.135325 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmwf\" (UniqueName: \"kubernetes.io/projected/1acd164c-032f-40eb-bef6-b07042848a90-kube-api-access-wfmwf\") pod \"node-ca-mj48w\" (UID: \"1acd164c-032f-40eb-bef6-b07042848a90\") " pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.135453 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.135359 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv25m\" (UniqueName: \"kubernetes.io/projected/dea71310-9b24-40e8-8b58-8780a189b3df-kube-api-access-tv25m\") pod \"multus-additional-cni-plugins-75jhn\" (UID: \"dea71310-9b24-40e8-8b58-8780a189b3df\") " pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.136257 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.136233 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kfw\" (UniqueName: \"kubernetes.io/projected/022e6ace-b742-4eb3-a13a-97d1ec1f10e3-kube-api-access-k2kfw\") pod \"node-resolver-tp929\" (UID: \"022e6ace-b742-4eb3-a13a-97d1ec1f10e3\") " pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.136354 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.136291 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcj26\" (UniqueName: \"kubernetes.io/projected/06b28783-48ec-4b28-94de-385cd15baaa8-kube-api-access-zcj26\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:37.136709 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.136671 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzks7\" (UniqueName: \"kubernetes.io/projected/ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b-kube-api-access-mzks7\") pod \"aws-ebs-csi-driver-node-t647c\" (UID: \"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.137542 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.137523 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvdlx\" (UniqueName: \"kubernetes.io/projected/333ddf02-969b-40cf-8d27-bb408a5a3873-kube-api-access-vvdlx\") pod \"iptables-alerter-qdg7w\" (UID: \"333ddf02-969b-40cf-8d27-bb408a5a3873\") " pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.139925 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.139909 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:07:37.237838 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.237769 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qdg7w" Jul 28 10:07:37.250751 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.250727 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:07:37.267815 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.267794 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" Jul 28 10:07:37.271413 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.271383 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b32d16ed33041d1fceff9e2d07ef80.slice/crio-0a35c4159b7b51f96e22e4ff88c661adb8b06920d92c440e834bc77ea6c3d967 WatchSource:0}: Error finding container 0a35c4159b7b51f96e22e4ff88c661adb8b06920d92c440e834bc77ea6c3d967: Status 404 returned error can't find the container with id 0a35c4159b7b51f96e22e4ff88c661adb8b06920d92c440e834bc77ea6c3d967 Jul 28 10:07:37.271893 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.271872 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode564183a_fced_4212_9fcf_6c1ffbf65cfa.slice/crio-c5dbeb42f712fa31b4b7733dcb818dcca247d5ce148216f23085e9e7426cd85e WatchSource:0}: Error finding container c5dbeb42f712fa31b4b7733dcb818dcca247d5ce148216f23085e9e7426cd85e: Status 404 returned error can't find the container with id c5dbeb42f712fa31b4b7733dcb818dcca247d5ce148216f23085e9e7426cd85e Jul 28 10:07:37.272612 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.272590 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333ddf02_969b_40cf_8d27_bb408a5a3873.slice/crio-3f284798a2632db70014f749a7527d02755869c4642ffe47caafeb39f944eb9e WatchSource:0}: Error finding container 3f284798a2632db70014f749a7527d02755869c4642ffe47caafeb39f944eb9e: Status 404 returned error can't find the container with id 3f284798a2632db70014f749a7527d02755869c4642ffe47caafeb39f944eb9e Jul 28 10:07:37.278189 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.277885 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 10:07:37.282304 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.282286 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mj48w" Jul 28 10:07:37.291054 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.291032 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acd164c_032f_40eb_bef6_b07042848a90.slice/crio-e122f3b2282741fe6f24a79f2053ba32be7a750baf6d5871be12d5e7f10efc26 WatchSource:0}: Error finding container e122f3b2282741fe6f24a79f2053ba32be7a750baf6d5871be12d5e7f10efc26: Status 404 returned error can't find the container with id e122f3b2282741fe6f24a79f2053ba32be7a750baf6d5871be12d5e7f10efc26 Jul 28 10:07:37.299038 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.299017 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z899d" Jul 28 10:07:37.319508 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.319484 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:07:37.319614 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.319540 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e04923af854eb13d5fa434fe913350.slice/crio-b9bd25ce615876ccba52e6b9605154ae2b2d29d82b93a7d73bfef357dced4ef6 WatchSource:0}: Error finding container b9bd25ce615876ccba52e6b9605154ae2b2d29d82b93a7d73bfef357dced4ef6: Status 404 returned error can't find the container with id b9bd25ce615876ccba52e6b9605154ae2b2d29d82b93a7d73bfef357dced4ef6 Jul 28 10:07:37.320143 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.320123 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7062085c_ecbf_476b_b845_e594877f19d9.slice/crio-b2f0a29ffaef0c4d5b15933de91de15a2de7fba791ba4637d97bb7acfbd8ee1e WatchSource:0}: Error finding container b2f0a29ffaef0c4d5b15933de91de15a2de7fba791ba4637d97bb7acfbd8ee1e: Status 404 returned error can't find the container with id b2f0a29ffaef0c4d5b15933de91de15a2de7fba791ba4637d97bb7acfbd8ee1e Jul 28 10:07:37.328955 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.328934 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f63461_f09b_4642_b556_023c9f371845.slice/crio-529d71a6e3e87dfb4e74f8ebfbbe68eb0d23cfbbba74e4d9415d5b3afdc3b0c7 WatchSource:0}: Error finding container 529d71a6e3e87dfb4e74f8ebfbbe68eb0d23cfbbba74e4d9415d5b3afdc3b0c7: Status 404 returned error can't find the container with id 529d71a6e3e87dfb4e74f8ebfbbe68eb0d23cfbbba74e4d9415d5b3afdc3b0c7 Jul 28 10:07:37.333483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.333460 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" Jul 28 10:07:37.342553 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.342532 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2c0932_05d0_49ca_a3f5_a4ae9ee5931b.slice/crio-065e61fdc4f2882f29ce0b2219082dc1a73b3ffdffbbe5c4a927e020b1fc1a8b WatchSource:0}: Error finding container 065e61fdc4f2882f29ce0b2219082dc1a73b3ffdffbbe5c4a927e020b1fc1a8b: Status 404 returned error can't find the container with id 065e61fdc4f2882f29ce0b2219082dc1a73b3ffdffbbe5c4a927e020b1fc1a8b Jul 28 10:07:37.348777 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.348753 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tp929" Jul 28 10:07:37.353912 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.353895 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-75jhn" Jul 28 10:07:37.358160 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.358136 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022e6ace_b742_4eb3_a13a_97d1ec1f10e3.slice/crio-1f848fdd1fea7d5ad15a718efecca034062f4c03fe552fe3292f0146bb9386aa WatchSource:0}: Error finding container 1f848fdd1fea7d5ad15a718efecca034062f4c03fe552fe3292f0146bb9386aa: Status 404 returned error can't find the container with id 1f848fdd1fea7d5ad15a718efecca034062f4c03fe552fe3292f0146bb9386aa Jul 28 10:07:37.362737 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:07:37.362719 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea71310_9b24_40e8_8b58_8780a189b3df.slice/crio-7b142e075636a88a36eb56af0319f8f2934dde8e90155585f66f17e1762a5484 WatchSource:0}: Error finding container 7b142e075636a88a36eb56af0319f8f2934dde8e90155585f66f17e1762a5484: Status 404 returned error can't find the container with id 7b142e075636a88a36eb56af0319f8f2934dde8e90155585f66f17e1762a5484 Jul 28 10:07:37.364258 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.364237 2569 kuberuntime_manager.go:1459] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-75jhn_openshift-multus(dea71310-9b24-40e8-8b58-8780a189b3df): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 28 10:07:37.364339 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.364268 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-75jhn" podUID="dea71310-9b24-40e8-8b58-8780a189b3df" Jul 28 10:07:37.622245 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.622143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:37.622245 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.622193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:37.622441 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.622301 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:37.622441 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.622304 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:37.622441 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.622362 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret podName:cbac8868-5bb5-4857-895f-455556932955 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:38.622346206 +0000 UTC m=+3.064925161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret") pod "global-pull-secret-syncer-q6r8g" (UID: "cbac8868-5bb5-4857-895f-455556932955") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:37.622441 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.622379 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs podName:06b28783-48ec-4b28-94de-385cd15baaa8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:38.622372345 +0000 UTC m=+3.064951300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs") pod "network-metrics-daemon-trrp4" (UID: "06b28783-48ec-4b28-94de-385cd15baaa8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:37.722820 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.722783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:37.722991 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.722961 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:07:37.722991 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.722989 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:07:37.723068 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.723001 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4vdgc for pod openshift-network-diagnostics/network-check-target-fr92k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:37.723068 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:37.723058 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc podName:d6b7677d-6c6c-4be6-893c-936e9ca050be nodeName:}" failed. No retries permitted until 2026-07-28 10:07:38.723043422 +0000 UTC m=+3.165622375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vdgc" (UniqueName: "kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc") pod "network-check-target-fr92k" (UID: "d6b7677d-6c6c-4be6-893c-936e9ca050be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:37.874521 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.874248 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:07:37.976018 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.975949 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:37 +0000 UTC" deadline="2028-05-10 23:22:05.180788484 +0000 UTC" Jul 28 10:07:37.976018 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:37.976016 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15661h14m27.204777192s" Jul 28 10:07:38.076260 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.076225 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:38.076470 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.076388 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:38.092580 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.092522 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z899d" event={"ID":"7062085c-ecbf-476b-b845-e594877f19d9","Type":"ContainerStarted","Data":"b2f0a29ffaef0c4d5b15933de91de15a2de7fba791ba4637d97bb7acfbd8ee1e"} Jul 28 10:07:38.104851 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.104816 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" event={"ID":"19bdd7fe-0327-4db4-83ad-38b5b334cbdc","Type":"ContainerStarted","Data":"52f770a16290855b87b723ef843541e61eefddbe9c7c6c4432788a91948dba3e"} Jul 28 10:07:38.112145 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.112113 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r4n9j" event={"ID":"e564183a-fced-4212-9fcf-6c1ffbf65cfa","Type":"ContainerStarted","Data":"c5dbeb42f712fa31b4b7733dcb818dcca247d5ce148216f23085e9e7426cd85e"} Jul 28 10:07:38.129183 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.129105 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qdg7w" event={"ID":"333ddf02-969b-40cf-8d27-bb408a5a3873","Type":"ContainerStarted","Data":"3f284798a2632db70014f749a7527d02755869c4642ffe47caafeb39f944eb9e"} Jul 28 10:07:38.135407 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.135375 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75jhn" event={"ID":"dea71310-9b24-40e8-8b58-8780a189b3df","Type":"ContainerStarted","Data":"7b142e075636a88a36eb56af0319f8f2934dde8e90155585f66f17e1762a5484"} Jul 28 10:07:38.143395 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.143364 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-75jhn" podUID="dea71310-9b24-40e8-8b58-8780a189b3df" Jul 28 10:07:38.160015 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.159981 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tp929" event={"ID":"022e6ace-b742-4eb3-a13a-97d1ec1f10e3","Type":"ContainerStarted","Data":"1f848fdd1fea7d5ad15a718efecca034062f4c03fe552fe3292f0146bb9386aa"} Jul 28 10:07:38.170556 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.170525 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" event={"ID":"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b","Type":"ContainerStarted","Data":"065e61fdc4f2882f29ce0b2219082dc1a73b3ffdffbbe5c4a927e020b1fc1a8b"} Jul 28 10:07:38.182259 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.182230 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" event={"ID":"33e04923af854eb13d5fa434fe913350","Type":"ContainerStarted","Data":"b9bd25ce615876ccba52e6b9605154ae2b2d29d82b93a7d73bfef357dced4ef6"} Jul 28 10:07:38.191210 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.191175 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mj48w" event={"ID":"1acd164c-032f-40eb-bef6-b07042848a90","Type":"ContainerStarted","Data":"e122f3b2282741fe6f24a79f2053ba32be7a750baf6d5871be12d5e7f10efc26"} Jul 28 10:07:38.200613 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.200549 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal" event={"ID":"05b32d16ed33041d1fceff9e2d07ef80","Type":"ContainerStarted","Data":"0a35c4159b7b51f96e22e4ff88c661adb8b06920d92c440e834bc77ea6c3d967"} Jul 28 10:07:38.207586 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.207557 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" event={"ID":"12f63461-f09b-4642-b556-023c9f371845","Type":"ContainerStarted","Data":"529d71a6e3e87dfb4e74f8ebfbbe68eb0d23cfbbba74e4d9415d5b3afdc3b0c7"} Jul 28 10:07:38.581839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.581756 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:07:38.630797 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.630760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:38.630990 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.630836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:38.631063 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.630991 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:38.631063 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.631056 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs podName:06b28783-48ec-4b28-94de-385cd15baaa8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:40.63103718 +0000 UTC m=+5.073616150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs") pod "network-metrics-daemon-trrp4" (UID: "06b28783-48ec-4b28-94de-385cd15baaa8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:38.631536 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.631450 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:38.631536 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.631500 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret podName:cbac8868-5bb5-4857-895f-455556932955 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:40.631485348 +0000 UTC m=+5.074064308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret") pod "global-pull-secret-syncer-q6r8g" (UID: "cbac8868-5bb5-4857-895f-455556932955") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:38.731635 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.731585 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:38.731829 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.731778 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:07:38.731829 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.731800 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:07:38.731829 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.731812 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4vdgc for pod openshift-network-diagnostics/network-check-target-fr92k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:38.732004 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:38.731885 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc podName:d6b7677d-6c6c-4be6-893c-936e9ca050be nodeName:}" failed. No retries permitted until 2026-07-28 10:07:40.731852264 +0000 UTC m=+5.174431241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vdgc" (UniqueName: "kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc") pod "network-check-target-fr92k" (UID: "d6b7677d-6c6c-4be6-893c-936e9ca050be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:38.976957 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.976858 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:02:37 +0000 UTC" deadline="2028-05-06 18:36:44.03885859 +0000 UTC" Jul 28 10:07:38.976957 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:38.976904 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15560h29m5.061960227s" Jul 28 10:07:39.073753 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:39.073694 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:39.074001 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:39.073849 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:39.074001 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:39.073953 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:39.074137 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:39.074071 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:39.202589 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:39.202555 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 10:07:40.073765 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:40.073727 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:40.074275 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:40.073893 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:40.650004 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:40.649908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:40.650154 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:40.650012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:40.650220 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:40.650176 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:40.650262 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:40.650246 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret podName:cbac8868-5bb5-4857-895f-455556932955 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:44.650223924 +0000 UTC m=+9.092802880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret") pod "global-pull-secret-syncer-q6r8g" (UID: "cbac8868-5bb5-4857-895f-455556932955") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:40.650766 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:40.650743 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:40.650880 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:40.650804 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs podName:06b28783-48ec-4b28-94de-385cd15baaa8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:44.650786894 +0000 UTC m=+9.093365851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs") pod "network-metrics-daemon-trrp4" (UID: "06b28783-48ec-4b28-94de-385cd15baaa8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:40.751163 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:40.751126 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:40.751356 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:40.751311 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:07:40.751356 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:40.751339 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:07:40.751356 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:40.751354 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4vdgc for pod openshift-network-diagnostics/network-check-target-fr92k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:40.751460 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:40.751427 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc podName:d6b7677d-6c6c-4be6-893c-936e9ca050be nodeName:}" failed. No retries permitted until 2026-07-28 10:07:44.751405405 +0000 UTC m=+9.193984377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vdgc" (UniqueName: "kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc") pod "network-check-target-fr92k" (UID: "d6b7677d-6c6c-4be6-893c-936e9ca050be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:41.073462 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:41.073369 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:41.073462 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:41.073419 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:41.073766 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:41.073513 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:41.073766 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:41.073599 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:42.073793 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:42.073759 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:42.073986 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:42.073908 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:43.074110 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:43.073370 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:43.074110 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:43.073508 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:43.074110 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:43.073959 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:43.074110 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:43.074059 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:44.076201 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:44.075718 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:44.076201 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:44.075869 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:44.683514 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:44.683152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:44.683514 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:44.683214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:44.683514 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:44.683322 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:44.683514 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:44.683346 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:44.683514 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:44.683393 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret podName:cbac8868-5bb5-4857-895f-455556932955 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:52.683375746 +0000 UTC m=+17.125954710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret") pod "global-pull-secret-syncer-q6r8g" (UID: "cbac8868-5bb5-4857-895f-455556932955") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:44.683514 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:44.683412 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs podName:06b28783-48ec-4b28-94de-385cd15baaa8 nodeName:}" failed. No retries permitted until 2026-07-28 10:07:52.683403579 +0000 UTC m=+17.125982545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs") pod "network-metrics-daemon-trrp4" (UID: "06b28783-48ec-4b28-94de-385cd15baaa8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:44.784517 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:44.784357 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:44.784749 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:44.784553 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:07:44.784749 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:44.784574 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:07:44.784749 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:44.784583 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4vdgc for pod openshift-network-diagnostics/network-check-target-fr92k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:44.784749 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:44.784661 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc podName:d6b7677d-6c6c-4be6-893c-936e9ca050be nodeName:}" failed. No retries permitted until 2026-07-28 10:07:52.784616314 +0000 UTC m=+17.227195269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vdgc" (UniqueName: "kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc") pod "network-check-target-fr92k" (UID: "d6b7677d-6c6c-4be6-893c-936e9ca050be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:45.073763 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:45.073681 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:45.073920 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:45.073814 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:45.074222 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:45.074203 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:45.074317 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:45.074300 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:46.074298 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:46.074263 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:46.074707 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:46.074412 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:47.073749 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:47.073716 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:47.073927 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:47.073843 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:47.073927 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:47.073907 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:47.074061 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:47.074021 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:48.073847 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:48.073815 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:48.074311 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:48.073959 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:49.073698 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:49.073654 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:49.073886 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:49.073654 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:49.073886 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:49.073817 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:49.073886 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:49.073876 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:50.074094 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:50.074057 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:50.074553 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:50.074219 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:51.073053 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:51.073011 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:51.073251 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:51.073012 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:51.073251 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:51.073147 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:51.073376 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:51.073249 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:52.073310 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:52.073277 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:52.073660 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:52.073396 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:52.745380 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:52.745345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:52.745574 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:52.745395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:52.745574 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:52.745514 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:52.745574 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:52.745512 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:52.745731 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:52.745576 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs podName:06b28783-48ec-4b28-94de-385cd15baaa8 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:08.745556481 +0000 UTC m=+33.188135435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs") pod "network-metrics-daemon-trrp4" (UID: "06b28783-48ec-4b28-94de-385cd15baaa8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:07:52.745731 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:52.745631 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret podName:cbac8868-5bb5-4857-895f-455556932955 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:08.74560137 +0000 UTC m=+33.188180323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret") pod "global-pull-secret-syncer-q6r8g" (UID: "cbac8868-5bb5-4857-895f-455556932955") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:07:52.846436 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:52.846391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:52.846647 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:52.846596 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:07:52.846647 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:52.846636 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:07:52.846647 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:52.846647 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4vdgc for pod openshift-network-diagnostics/network-check-target-fr92k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:52.846769 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:52.846699 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc podName:d6b7677d-6c6c-4be6-893c-936e9ca050be nodeName:}" failed. No retries permitted until 2026-07-28 10:08:08.846685134 +0000 UTC m=+33.289264087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vdgc" (UniqueName: "kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc") pod "network-check-target-fr92k" (UID: "d6b7677d-6c6c-4be6-893c-936e9ca050be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:07:53.074150 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:53.074069 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:53.074150 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:53.074115 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:53.074579 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:53.074212 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:53.074579 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:53.074329 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:54.073883 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:54.073852 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:54.074061 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:54.073988 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:55.073639 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:55.073597 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:55.073639 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:55.073612 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:55.074092 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:55.073738 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:55.074092 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:55.073877 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:56.074188 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:56.074152 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:56.074561 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:56.074279 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:57.074090 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.073702 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:57.074249 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.073736 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:57.074249 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:57.074182 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:57.075194 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:57.074263 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:57.250086 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.250046 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tp929" event={"ID":"022e6ace-b742-4eb3-a13a-97d1ec1f10e3","Type":"ContainerStarted","Data":"ec0be3068417907fa69d690bbc8b75e6a09b8e9ee7195b0305911cea3e1bb053"} Jul 28 10:07:57.251660 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.251634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" event={"ID":"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b","Type":"ContainerStarted","Data":"5534663a281114c5a3314a2620b2279a6f39cebf025b8b137e17f17ce25c5727"} Jul 28 10:07:57.253405 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.253380 2569 generic.go:356] "Generic (PLEG): container finished" podID="33e04923af854eb13d5fa434fe913350" containerID="593a2fcf50a9f9d8fa7a5061c0c86471bcd1703fba12efc9d327042317ba2b7b" exitCode=0 Jul 28 10:07:57.253515 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.253464 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" event={"ID":"33e04923af854eb13d5fa434fe913350","Type":"ContainerDied","Data":"593a2fcf50a9f9d8fa7a5061c0c86471bcd1703fba12efc9d327042317ba2b7b"} Jul 28 10:07:57.255130 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.255102 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mj48w" event={"ID":"1acd164c-032f-40eb-bef6-b07042848a90","Type":"ContainerStarted","Data":"d81328a1288c8e227a3da86a6489199a7275015d966066f0f9c875d14b849061"} Jul 28 10:07:57.257015 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.256992 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal" event={"ID":"05b32d16ed33041d1fceff9e2d07ef80","Type":"ContainerStarted","Data":"f3a838a7352d41c0636c70bec336cc6b689cdfad23ecac33569484593cc1715e"} Jul 28 10:07:57.260492 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.260468 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" event={"ID":"12f63461-f09b-4642-b556-023c9f371845","Type":"ContainerStarted","Data":"c45e15f6f10526ac22290da778b329723b98ffdde838fea476bc17d6cba4dd96"} Jul 28 10:07:57.260571 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.260537 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" event={"ID":"12f63461-f09b-4642-b556-023c9f371845","Type":"ContainerStarted","Data":"02e73129c5b19754ab7af03e633d3468f2593587a3fe00191cf3113c46e49440"} Jul 28 10:07:57.260637 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.260580 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" event={"ID":"12f63461-f09b-4642-b556-023c9f371845","Type":"ContainerStarted","Data":"fa2376cc2747474f7a5b0a2e7f3f9bf02b232364a5a5cba6d0cea59393a131da"} Jul 28 10:07:57.260637 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.260608 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" event={"ID":"12f63461-f09b-4642-b556-023c9f371845","Type":"ContainerStarted","Data":"f17d7f560164fb64c58f15016053ede1cc58357d3d4f01d1b7a71d1e37fcf8b4"} Jul 28 10:07:57.260709 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.260661 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" event={"ID":"12f63461-f09b-4642-b556-023c9f371845","Type":"ContainerStarted","Data":"ebcc49a9daabf7b5598b253fa95a86064450b93d35b4fb1c073a9a1030494e28"} Jul 28 10:07:57.260754 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.260708 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" event={"ID":"12f63461-f09b-4642-b556-023c9f371845","Type":"ContainerStarted","Data":"8d7bd6dd92f739262d635715ddf8848176dd5cce575c0c2b940baec1db263854"} Jul 28 10:07:57.263068 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.262337 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z899d" event={"ID":"7062085c-ecbf-476b-b845-e594877f19d9","Type":"ContainerStarted","Data":"6bc1c3a8ede4d3ed527ef5b000d5aa729491ea4194e50669e622720f0ee8eb17"} Jul 28 10:07:57.264406 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.264380 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" event={"ID":"19bdd7fe-0327-4db4-83ad-38b5b334cbdc","Type":"ContainerStarted","Data":"46ed06baf5cee2cdb42ecb48a306eb8c502645e2f30b084af00f3ff2307b2ef4"} Jul 28 10:07:57.266900 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.266863 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r4n9j" event={"ID":"e564183a-fced-4212-9fcf-6c1ffbf65cfa","Type":"ContainerStarted","Data":"00262d2a1b4a0e358fef501d6f783c4c06d24f28f079921a8693aa176049640b"} Jul 28 10:07:57.268162 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.268123 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tp929" podStartSLOduration=2.436510782 podStartE2EDuration="21.268109107s" totalImagesPullingTime="18.831598325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:57.267524546 +0000 UTC m=+21.710103522" watchObservedRunningTime="2026-07-28 10:07:57.268109107 +0000 UTC m=+21.710688083" Jul 28 10:07:57.283710 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.283652 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mj48w" podStartSLOduration=2.386128454 podStartE2EDuration="21.28363624s" totalImagesPullingTime="18.897507786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:57.283359431 +0000 UTC m=+21.725938406" watchObservedRunningTime="2026-07-28 10:07:57.28363624 +0000 UTC m=+21.726215214" Jul 28 10:07:57.303773 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.303698 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z899d" podStartSLOduration=2.278922558 podStartE2EDuration="21.303677195s" totalImagesPullingTime="19.024754637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:57.303363816 +0000 UTC m=+21.745942790" watchObservedRunningTime="2026-07-28 10:07:57.303677195 +0000 UTC m=+21.746256171" Jul 28 10:07:57.323724 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.323671 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-126.ec2.internal" podStartSLOduration=20.323650218 podStartE2EDuration="20.323650218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:57.322805841 +0000 UTC m=+21.765384816" watchObservedRunningTime="2026-07-28 10:07:57.323650218 +0000 UTC m=+21.766229203" Jul 28 10:07:57.351943 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.351885 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b2j8p" podStartSLOduration=2.419147573 podStartE2EDuration="21.351864496s" totalImagesPullingTime="18.932716923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:57.350925517 +0000 UTC m=+21.793504492" watchObservedRunningTime="2026-07-28 10:07:57.351864496 +0000 UTC m=+21.794443472" Jul 28 10:07:57.408017 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:57.407961 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r4n9j" podStartSLOduration=6.673891431 podStartE2EDuration="21.407939236s" totalImagesPullingTime="14.734047805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:57.40764381 +0000 UTC m=+21.850222785" watchObservedRunningTime="2026-07-28 10:07:57.407939236 +0000 UTC m=+21.850518211" Jul 28 10:07:58.073926 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:58.073890 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:07:58.074113 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:58.074037 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:07:58.269918 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:58.269879 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qdg7w" event={"ID":"333ddf02-969b-40cf-8d27-bb408a5a3873","Type":"ContainerStarted","Data":"60c150cdc067c16127f5592c0f2df4951d35244885cdcc496dc471bbeba821ea"} Jul 28 10:07:58.287976 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:58.287934 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qdg7w" podStartSLOduration=3.354645231 podStartE2EDuration="22.287919163s" totalImagesPullingTime="18.933273932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:58.287786304 +0000 UTC m=+22.730365278" watchObservedRunningTime="2026-07-28 10:07:58.287919163 +0000 UTC m=+22.730498137" Jul 28 10:07:58.442643 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:58.442351 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 10:07:58.985675 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:58.985574 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T10:07:58.442590715Z","UUID":"c3a2e2bd-ce14-47fb-b018-dd439ff7f002","Handler":null,"Name":"","Endpoint":""} Jul 28 10:07:58.987125 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:58.987107 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 10:07:58.987229 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:58.987138 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 10:07:59.073256 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:59.073235 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:07:59.073347 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:59.073329 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:07:59.073432 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:59.073379 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:07:59.073469 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:07:59.073456 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:07:59.273902 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:59.273867 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" event={"ID":"33e04923af854eb13d5fa434fe913350","Type":"ContainerStarted","Data":"ba999f026786b66c5245b92a4ee38f2df0ff99fa3c257cacc0a3830927b6acfc"} Jul 28 10:07:59.277095 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:59.277071 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" event={"ID":"12f63461-f09b-4642-b556-023c9f371845","Type":"ContainerStarted","Data":"0f999a0e13427b46715897980c958180a0f0944606b2403671fc9f1e6e29370a"} Jul 28 10:07:59.278530 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:59.278507 2569 generic.go:356] "Generic (PLEG): container finished" podID="dea71310-9b24-40e8-8b58-8780a189b3df" containerID="73d30f12038d09c09713359385ab1a07828fd5e4ab575124c0d3ef5d5ba329c8" exitCode=0 Jul 28 10:07:59.278650 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:59.278570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75jhn" event={"ID":"dea71310-9b24-40e8-8b58-8780a189b3df","Type":"ContainerDied","Data":"73d30f12038d09c09713359385ab1a07828fd5e4ab575124c0d3ef5d5ba329c8"} Jul 28 10:07:59.280331 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:59.280308 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" event={"ID":"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b","Type":"ContainerStarted","Data":"445f73a209ddfeaf6f6acd047aaec2d36dd27d48d4e00a7a37b723a432c93cf2"} Jul 28 10:07:59.295555 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:07:59.295513 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-126.ec2.internal" podStartSLOduration=22.295499987 podStartE2EDuration="22.295499987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:07:59.294736362 +0000 UTC m=+23.737315336" watchObservedRunningTime="2026-07-28 10:07:59.295499987 +0000 UTC m=+23.738078964" Jul 28 10:08:00.073136 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:00.073101 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:00.073280 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:00.073238 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:08:00.285085 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:00.285046 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" event={"ID":"ea2c0932-05d0-49ca-a3f5-a4ae9ee5931b","Type":"ContainerStarted","Data":"3964d053ef0de686b15a448d0830318e8b8794ec5364e12ea4cd1235a0b4b956"} Jul 28 10:08:00.306424 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:00.306374 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t647c" podStartSLOduration=2.26995717 podStartE2EDuration="24.306358516s" totalImagesPullingTime="22.036401346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:00.306069591 +0000 UTC m=+24.748648593" watchObservedRunningTime="2026-07-28 10:08:00.306358516 +0000 UTC m=+24.748937506" Jul 28 10:08:01.073417 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:01.073385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:01.073609 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:01.073385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:01.073609 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:01.073529 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:08:01.073756 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:01.073615 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:08:01.295836 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:01.295576 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" event={"ID":"12f63461-f09b-4642-b556-023c9f371845","Type":"ContainerStarted","Data":"599bec4f728ca3f84d4dcc21d54f4462c4c07ae108cf56e4bfb2f3d338007cf0"} Jul 28 10:08:01.761791 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:01.761753 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:08:01.762485 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:01.762462 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:08:02.074019 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:02.073934 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:02.074149 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:02.074086 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:08:02.298899 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:02.298857 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:08:02.299548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:02.299378 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r4n9j" Jul 28 10:08:02.328775 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:02.328663 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" podStartSLOduration=7.394387504 podStartE2EDuration="26.328644406s" totalImagesPullingTime="18.934256902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:02.326966762 +0000 UTC m=+26.769545738" watchObservedRunningTime="2026-07-28 10:08:02.328644406 +0000 UTC m=+26.771223383" Jul 28 10:08:03.074061 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.073971 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:03.074224 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.073971 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:03.074224 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:03.074106 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:08:03.074224 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:03.074179 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:08:03.300983 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.300953 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:08:03.301616 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.301017 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:08:03.301616 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.301047 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:08:03.315918 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.315893 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:08:03.316187 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.316170 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:08:03.335776 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.335689 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-trrp4"] Jul 28 10:08:03.335925 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.335800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:03.335925 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:03.335905 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:08:03.339370 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.339336 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fr92k"] Jul 28 10:08:03.339512 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.339433 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:03.339569 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:03.339533 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:08:03.340066 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.340045 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q6r8g"] Jul 28 10:08:03.340156 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:03.340126 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:03.340240 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:03.340221 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:08:04.303786 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:04.303577 2569 generic.go:356] "Generic (PLEG): container finished" podID="dea71310-9b24-40e8-8b58-8780a189b3df" containerID="02c0ca8a5c6619578e4d4069f210ef9975180b2038f513d4e0da6a948347859a" exitCode=0 Jul 28 10:08:04.304132 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:04.303651 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75jhn" event={"ID":"dea71310-9b24-40e8-8b58-8780a189b3df","Type":"ContainerDied","Data":"02c0ca8a5c6619578e4d4069f210ef9975180b2038f513d4e0da6a948347859a"} Jul 28 10:08:05.073972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:05.073946 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:05.074093 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:05.074002 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:05.074133 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:05.074096 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:08:05.074249 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:05.074231 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:08:05.074284 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:05.074258 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:05.074350 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:05.074319 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:08:05.308786 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:05.308746 2569 generic.go:356] "Generic (PLEG): container finished" podID="dea71310-9b24-40e8-8b58-8780a189b3df" containerID="cc3d163490cdbf17fe0c43f21e3cdb3b10ce3bd6cea9225d7fee36cc25a8bd2d" exitCode=0 Jul 28 10:08:05.309234 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:05.308815 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75jhn" event={"ID":"dea71310-9b24-40e8-8b58-8780a189b3df","Type":"ContainerDied","Data":"cc3d163490cdbf17fe0c43f21e3cdb3b10ce3bd6cea9225d7fee36cc25a8bd2d"} Jul 28 10:08:07.073598 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:07.073562 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:07.073957 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:07.073678 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:07.073957 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:07.073688 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:08:07.073957 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:07.073707 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:07.073957 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:07.073775 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:08:07.073957 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:07.073839 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:08:07.315019 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:07.314989 2569 generic.go:356] "Generic (PLEG): container finished" podID="dea71310-9b24-40e8-8b58-8780a189b3df" containerID="ddcdc9a0f88c308d7bf1dc4217bf2f27a2565fad088925d8cf9246b7ded9fc49" exitCode=0 Jul 28 10:08:07.315155 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:07.315048 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75jhn" event={"ID":"dea71310-9b24-40e8-8b58-8780a189b3df","Type":"ContainerDied","Data":"ddcdc9a0f88c308d7bf1dc4217bf2f27a2565fad088925d8cf9246b7ded9fc49"} Jul 28 10:08:08.773135 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:08.772908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:08.773575 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:08.773159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:08.773575 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:08.773060 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 10:08:08.773575 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:08.773263 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret podName:cbac8868-5bb5-4857-895f-455556932955 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:40.77324523 +0000 UTC m=+65.215824200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret") pod "global-pull-secret-syncer-q6r8g" (UID: "cbac8868-5bb5-4857-895f-455556932955") : object "kube-system"/"original-pull-secret" not registered Jul 28 10:08:08.773575 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:08.773280 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:08:08.773575 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:08.773314 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs podName:06b28783-48ec-4b28-94de-385cd15baaa8 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:40.773302778 +0000 UTC m=+65.215881747 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs") pod "network-metrics-daemon-trrp4" (UID: "06b28783-48ec-4b28-94de-385cd15baaa8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 10:08:08.874461 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:08.874429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:08.874651 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:08.874567 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 10:08:08.874651 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:08.874589 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 10:08:08.874651 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:08.874603 2569 projected.go:196] Error preparing data for projected volume kube-api-access-4vdgc for pod openshift-network-diagnostics/network-check-target-fr92k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:08:08.874827 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:08.874679 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc podName:d6b7677d-6c6c-4be6-893c-936e9ca050be nodeName:}" failed. No retries permitted until 2026-07-28 10:08:40.874660501 +0000 UTC m=+65.317239475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4vdgc" (UniqueName: "kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc") pod "network-check-target-fr92k" (UID: "d6b7677d-6c6c-4be6-893c-936e9ca050be") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 10:08:09.073876 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.073800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:09.074028 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.073801 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:09.074028 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:09.073914 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fr92k" podUID="d6b7677d-6c6c-4be6-893c-936e9ca050be" Jul 28 10:08:09.074028 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:09.073978 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6r8g" podUID="cbac8868-5bb5-4857-895f-455556932955" Jul 28 10:08:09.074028 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.073800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:09.074190 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:09.074071 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-trrp4" podUID="06b28783-48ec-4b28-94de-385cd15baaa8" Jul 28 10:08:09.927847 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.927821 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-126.ec2.internal" event="NodeReady" Jul 28 10:08:09.928264 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.927966 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 10:08:09.971600 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.971566 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-659cc6d558-mxqwp"] Jul 28 10:08:09.985717 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.985688 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m"] Jul 28 10:08:09.985874 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.985816 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:09.989325 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.989305 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 10:08:09.989480 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.989326 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8x85w\"" Jul 28 10:08:09.990060 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.990045 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 10:08:09.990191 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.990168 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 10:08:09.997833 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.997816 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-jd9hz"] Jul 28 10:08:09.997952 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:09.997935 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.000159 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.000141 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 10:08:10.004548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.003272 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 28 10:08:10.004548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.003601 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 28 10:08:10.004548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.003920 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 28 10:08:10.004548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.004127 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:08:10.004548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.004362 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6p4cr\"" Jul 28 10:08:10.020265 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.020239 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw"] Jul 28 10:08:10.020408 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.020363 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.028456 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.028437 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-45s4n\"" Jul 28 10:08:10.028699 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.028684 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 10:08:10.029005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.028985 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 10:08:10.029573 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.029552 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 28 10:08:10.029858 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.029737 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 28 10:08:10.031183 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.031152 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87"] Jul 28 10:08:10.031470 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.031452 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw" Jul 28 10:08:10.035357 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.035331 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 28 10:08:10.035357 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.035344 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:08:10.035632 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.035606 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vfz86\"" Jul 28 10:08:10.035708 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.035678 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 28 10:08:10.045323 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.045301 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr"] Jul 28 10:08:10.045420 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.045402 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.049822 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.049801 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 10:08:10.050126 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.050039 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 10:08:10.050126 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.050117 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 10:08:10.050386 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.050369 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rf979\"" Jul 28 10:08:10.050457 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.050414 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 10:08:10.059795 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.059776 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-wkm97"] Jul 28 10:08:10.059913 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.059899 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.062872 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.062852 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 28 10:08:10.063124 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.063109 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:08:10.063205 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.063178 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9s4bh\"" Jul 28 10:08:10.063500 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.063480 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 28 10:08:10.063597 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.063576 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 28 10:08:10.070204 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.070186 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6b5bf78f8f-wkjq5"] Jul 28 10:08:10.070300 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.070289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.074464 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.074446 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 28 10:08:10.074654 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.074616 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 28 10:08:10.074747 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.074676 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q8th5\"" Jul 28 10:08:10.074960 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.074946 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 28 10:08:10.075248 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.075158 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:08:10.080374 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.080354 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 28 10:08:10.080669 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.080649 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.083389 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.083368 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 28 10:08:10.084050 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.084022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5psnq\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-kube-api-access-5psnq\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.084141 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.084121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-image-registry-private-configuration\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.085014 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.084031 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4zl7s\"" Jul 28 10:08:10.085014 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.084433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/977426ec-297c-41ec-ab46-c83d7cf4d480-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rsw5m\" (UID: \"977426ec-297c-41ec-ab46-c83d7cf4d480\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.085014 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.084546 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-certificates\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.085014 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.084704 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 28 10:08:10.085014 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.084723 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03330411-ab4f-43a6-92f3-cefc7526cc9f-ca-trust-extracted\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.085014 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.084810 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-bound-sa-token\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.085329 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.084958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.085329 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.085201 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swr5\" (UniqueName: \"kubernetes.io/projected/977426ec-297c-41ec-ab46-c83d7cf4d480-kube-api-access-9swr5\") pod \"service-ca-operator-5d6c75f84f-rsw5m\" (UID: \"977426ec-297c-41ec-ab46-c83d7cf4d480\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.085422 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.085349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/977426ec-297c-41ec-ab46-c83d7cf4d480-config\") pod \"service-ca-operator-5d6c75f84f-rsw5m\" (UID: \"977426ec-297c-41ec-ab46-c83d7cf4d480\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.085562 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.085448 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-installation-pull-secrets\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.085667 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.085585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-trusted-ca\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.085846 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.085830 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 10:08:10.087503 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.086263 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 28 10:08:10.087503 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.086309 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 10:08:10.087503 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.086512 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 28 10:08:10.089542 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.088794 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx"] Jul 28 10:08:10.099024 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.099005 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m"] Jul 28 10:08:10.099127 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.099038 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659cc6d558-mxqwp"] Jul 28 10:08:10.099127 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.099066 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw"] Jul 28 10:08:10.099127 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.099084 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87"] Jul 28 10:08:10.099127 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.099098 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-jd9hz"] Jul 28 10:08:10.099311 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.099193 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:10.099311 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.099203 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr"] Jul 28 10:08:10.099311 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.099222 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gdk8g"] Jul 28 10:08:10.102479 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.102462 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 28 10:08:10.102754 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.102517 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ndrbx\"" Jul 28 10:08:10.102852 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.102759 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 28 10:08:10.102980 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.102965 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 28 10:08:10.107351 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.107334 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b59sc"] Jul 28 10:08:10.107480 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.107455 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.111414 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.111397 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 10:08:10.111500 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.111489 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5fpr8\"" Jul 28 10:08:10.111543 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.111497 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 10:08:10.115977 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.115960 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx"] Jul 28 10:08:10.116088 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.116074 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b5bf78f8f-wkjq5"] Jul 28 10:08:10.116154 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.116114 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-wkm97"] Jul 28 10:08:10.116154 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.116141 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b59sc"] Jul 28 10:08:10.116240 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.116160 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gdk8g"] Jul 28 10:08:10.116240 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.116191 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b59sc" Jul 28 10:08:10.120661 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.120642 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 10:08:10.120754 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.120722 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 10:08:10.121222 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.121207 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mbb65\"" Jul 28 10:08:10.121396 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.121212 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 10:08:10.186438 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186402 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-default-certificate\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.186438 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186441 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82k7q\" (UniqueName: \"kubernetes.io/projected/17049657-4f76-49fa-b85c-d47d4bc68493-kube-api-access-82k7q\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:10.186713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186507 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnqhl\" (UniqueName: \"kubernetes.io/projected/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-kube-api-access-nnqhl\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.186713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-trusted-ca\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.186713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psnq\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-kube-api-access-5psnq\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.186713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ba36116-0fd9-4d91-b070-c52a5378dc54-serving-cert\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.186713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.186713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186662 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4cwk\" (UniqueName: \"kubernetes.io/projected/fafae72a-446d-4852-9799-ed64ff1569fd-kube-api-access-v4cwk\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.186713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186707 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/977426ec-297c-41ec-ab46-c83d7cf4d480-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rsw5m\" (UID: \"977426ec-297c-41ec-ab46-c83d7cf4d480\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-stats-auth\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186761 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ba36116-0fd9-4d91-b070-c52a5378dc54-tmp\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186789 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkksz\" (UniqueName: \"kubernetes.io/projected/6ba36116-0fd9-4d91-b070-c52a5378dc54-kube-api-access-wkksz\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm5qg\" (UniqueName: \"kubernetes.io/projected/25239465-8177-4ac4-9cb3-a293eb83589e-kube-api-access-bm5qg\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr\" (UID: \"25239465-8177-4ac4-9cb3-a293eb83589e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn4fl\" (UniqueName: \"kubernetes.io/projected/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-kube-api-access-cn4fl\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186874 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03330411-ab4f-43a6-92f3-cefc7526cc9f-ca-trust-extracted\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ba36116-0fd9-4d91-b070-c52a5378dc54-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ba36116-0fd9-4d91-b070-c52a5378dc54-service-ca-bundle\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186948 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fafae72a-446d-4852-9799-ed64ff1569fd-config\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.187032 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.186993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swr5\" (UniqueName: \"kubernetes.io/projected/977426ec-297c-41ec-ab46-c83d7cf4d480-kube-api-access-9swr5\") pod \"service-ca-operator-5d6c75f84f-rsw5m\" (UID: \"977426ec-297c-41ec-ab46-c83d7cf4d480\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.187487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187093 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6ba36116-0fd9-4d91-b070-c52a5378dc54-snapshots\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.187487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187130 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-installation-pull-secrets\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.187487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187135 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03330411-ab4f-43a6-92f3-cefc7526cc9f-ca-trust-extracted\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.187487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgj8\" (UniqueName: \"kubernetes.io/projected/056218d8-cf3f-4415-99c2-cac7b557f301-kube-api-access-pcgj8\") pod \"volume-data-source-validator-549c6d54cf-npsmw\" (UID: \"056218d8-cf3f-4415-99c2-cac7b557f301\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw" Jul 28 10:08:10.187487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187284 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.187487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187346 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/25239465-8177-4ac4-9cb3-a293eb83589e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr\" (UID: \"25239465-8177-4ac4-9cb3-a293eb83589e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.187487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187373 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.187487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-image-registry-private-configuration\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.187487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187447 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fafae72a-446d-4852-9799-ed64ff1569fd-serving-cert\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187493 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25239465-8177-4ac4-9cb3-a293eb83589e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr\" (UID: \"25239465-8177-4ac4-9cb3-a293eb83589e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-certificates\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187563 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-trusted-ca\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-bound-sa-token\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187656 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187704 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/977426ec-297c-41ec-ab46-c83d7cf4d480-config\") pod \"service-ca-operator-5d6c75f84f-rsw5m\" (UID: \"977426ec-297c-41ec-ab46-c83d7cf4d480\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.187725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fafae72a-446d-4852-9799-ed64ff1569fd-trusted-ca\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.187869 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 10:08:10.187889 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.187884 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659cc6d558-mxqwp: secret "image-registry-tls" not found Jul 28 10:08:10.188303 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.187932 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls podName:03330411-ab4f-43a6-92f3-cefc7526cc9f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:10.687915791 +0000 UTC m=+35.130494758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls") pod "image-registry-659cc6d558-mxqwp" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f") : secret "image-registry-tls" not found Jul 28 10:08:10.188303 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.188199 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/977426ec-297c-41ec-ab46-c83d7cf4d480-config\") pod \"service-ca-operator-5d6c75f84f-rsw5m\" (UID: \"977426ec-297c-41ec-ab46-c83d7cf4d480\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.188511 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.188486 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-certificates\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.191596 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.191570 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/977426ec-297c-41ec-ab46-c83d7cf4d480-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rsw5m\" (UID: \"977426ec-297c-41ec-ab46-c83d7cf4d480\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.191738 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.191712 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-image-registry-private-configuration\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.191738 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.191723 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-installation-pull-secrets\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.213209 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.213186 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-bound-sa-token\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.220165 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.220138 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swr5\" (UniqueName: \"kubernetes.io/projected/977426ec-297c-41ec-ab46-c83d7cf4d480-kube-api-access-9swr5\") pod \"service-ca-operator-5d6c75f84f-rsw5m\" (UID: \"977426ec-297c-41ec-ab46-c83d7cf4d480\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.225159 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.225117 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psnq\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-kube-api-access-5psnq\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.288605 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.288529 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:10.288605 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.288594 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fafae72a-446d-4852-9799-ed64ff1569fd-trusted-ca\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.288843 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.288637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-default-certificate\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.288843 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.288665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82k7q\" (UniqueName: \"kubernetes.io/projected/17049657-4f76-49fa-b85c-d47d4bc68493-kube-api-access-82k7q\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:10.288843 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.288691 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 10:08:10.288843 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.288709 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnqhl\" (UniqueName: \"kubernetes.io/projected/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-kube-api-access-nnqhl\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.288843 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.288739 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.288843 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.288768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ba36116-0fd9-4d91-b070-c52a5378dc54-serving-cert\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.288843 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.288816 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls podName:17049657-4f76-49fa-b85c-d47d4bc68493 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:10.788796049 +0000 UTC m=+35.231375015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls") pod "cluster-samples-operator-6748489656-t54tx" (UID: "17049657-4f76-49fa-b85c-d47d4bc68493") : secret "samples-operator-tls" not found Jul 28 10:08:10.288843 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.288834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.288941 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:10.788923308 +0000 UTC m=+35.231502287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : configmap references non-existent config key: service-ca.crt Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.288969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4cwk\" (UniqueName: \"kubernetes.io/projected/fafae72a-446d-4852-9799-ed64ff1569fd-kube-api-access-v4cwk\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-stats-auth\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ba36116-0fd9-4d91-b070-c52a5378dc54-tmp\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkksz\" (UniqueName: \"kubernetes.io/projected/6ba36116-0fd9-4d91-b070-c52a5378dc54-kube-api-access-wkksz\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8ba7d9-892e-4f45-846e-9d56500b25de-config-volume\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5qg\" (UniqueName: \"kubernetes.io/projected/25239465-8177-4ac4-9cb3-a293eb83589e-kube-api-access-bm5qg\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr\" (UID: \"25239465-8177-4ac4-9cb3-a293eb83589e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289126 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn4fl\" (UniqueName: \"kubernetes.io/projected/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-kube-api-access-cn4fl\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8ba7d9-892e-4f45-846e-9d56500b25de-tmp-dir\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxp7\" (UniqueName: \"kubernetes.io/projected/c1bf9f83-7660-43f9-8827-79237d176fcb-kube-api-access-tlxp7\") pod \"ingress-canary-b59sc\" (UID: \"c1bf9f83-7660-43f9-8827-79237d176fcb\") " pod="openshift-ingress-canary/ingress-canary-b59sc" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ba36116-0fd9-4d91-b070-c52a5378dc54-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ba36116-0fd9-4d91-b070-c52a5378dc54-service-ca-bundle\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.289286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289282 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fafae72a-446d-4852-9799-ed64ff1569fd-config\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6ba36116-0fd9-4d91-b070-c52a5378dc54-snapshots\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.289400 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgj8\" (UniqueName: \"kubernetes.io/projected/056218d8-cf3f-4415-99c2-cac7b557f301-kube-api-access-pcgj8\") pod \"volume-data-source-validator-549c6d54cf-npsmw\" (UID: \"056218d8-cf3f-4415-99c2-cac7b557f301\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289435 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.289442 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:10.789428703 +0000 UTC m=+35.232007669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : secret "router-metrics-certs-default" not found Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/25239465-8177-4ac4-9cb3-a293eb83589e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr\" (UID: \"25239465-8177-4ac4-9cb3-a293eb83589e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fafae72a-446d-4852-9799-ed64ff1569fd-serving-cert\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289566 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwm6\" (UniqueName: \"kubernetes.io/projected/5e8ba7d9-892e-4f45-846e-9d56500b25de-kube-api-access-hfwm6\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289605 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ba36116-0fd9-4d91-b070-c52a5378dc54-tmp\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289613 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25239465-8177-4ac4-9cb3-a293eb83589e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr\" (UID: \"25239465-8177-4ac4-9cb3-a293eb83589e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289675 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fafae72a-446d-4852-9799-ed64ff1569fd-trusted-ca\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289692 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1bf9f83-7660-43f9-8827-79237d176fcb-cert\") pod \"ingress-canary-b59sc\" (UID: \"c1bf9f83-7660-43f9-8827-79237d176fcb\") " pod="openshift-ingress-canary/ingress-canary-b59sc" Jul 28 10:08:10.290005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.289947 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ba36116-0fd9-4d91-b070-c52a5378dc54-service-ca-bundle\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.290821 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.290031 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:10.290821 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.290069 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fafae72a-446d-4852-9799-ed64ff1569fd-config\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.290821 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.290087 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls podName:fcdc3068-bf6d-437d-9b29-1210b9eb6bd6 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:10.790073245 +0000 UTC m=+35.232652198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-s6m87" (UID: "fcdc3068-bf6d-437d-9b29-1210b9eb6bd6") : secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:10.290821 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.290368 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.290821 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.290656 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/25239465-8177-4ac4-9cb3-a293eb83589e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr\" (UID: \"25239465-8177-4ac4-9cb3-a293eb83589e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.290821 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.290679 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6ba36116-0fd9-4d91-b070-c52a5378dc54-snapshots\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.291322 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.291300 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ba36116-0fd9-4d91-b070-c52a5378dc54-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.291882 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.291852 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-stats-auth\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.291971 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.291942 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ba36116-0fd9-4d91-b070-c52a5378dc54-serving-cert\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.292321 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.292303 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fafae72a-446d-4852-9799-ed64ff1569fd-serving-cert\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.292525 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.292506 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-default-certificate\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.297961 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.297932 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25239465-8177-4ac4-9cb3-a293eb83589e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr\" (UID: \"25239465-8177-4ac4-9cb3-a293eb83589e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.298286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.298269 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4cwk\" (UniqueName: \"kubernetes.io/projected/fafae72a-446d-4852-9799-ed64ff1569fd-kube-api-access-v4cwk\") pod \"console-operator-584dcb6b54-wkm97\" (UID: \"fafae72a-446d-4852-9799-ed64ff1569fd\") " pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.300942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.300899 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn4fl\" (UniqueName: \"kubernetes.io/projected/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-kube-api-access-cn4fl\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.301704 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.301678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5qg\" (UniqueName: \"kubernetes.io/projected/25239465-8177-4ac4-9cb3-a293eb83589e-kube-api-access-bm5qg\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr\" (UID: \"25239465-8177-4ac4-9cb3-a293eb83589e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.301859 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.301838 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkksz\" (UniqueName: \"kubernetes.io/projected/6ba36116-0fd9-4d91-b070-c52a5378dc54-kube-api-access-wkksz\") pod \"insights-operator-5885fdb759-jd9hz\" (UID: \"6ba36116-0fd9-4d91-b070-c52a5378dc54\") " pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.301921 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.301900 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnqhl\" (UniqueName: \"kubernetes.io/projected/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-kube-api-access-nnqhl\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.301973 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.301898 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgj8\" (UniqueName: \"kubernetes.io/projected/056218d8-cf3f-4415-99c2-cac7b557f301-kube-api-access-pcgj8\") pod \"volume-data-source-validator-549c6d54cf-npsmw\" (UID: \"056218d8-cf3f-4415-99c2-cac7b557f301\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw" Jul 28 10:08:10.302527 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.302505 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82k7q\" (UniqueName: \"kubernetes.io/projected/17049657-4f76-49fa-b85c-d47d4bc68493-kube-api-access-82k7q\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:10.314732 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.314710 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" Jul 28 10:08:10.334499 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.334468 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-jd9hz" Jul 28 10:08:10.342653 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.342613 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw" Jul 28 10:08:10.368674 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.368647 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" Jul 28 10:08:10.380312 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.380289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:10.390757 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.390735 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwm6\" (UniqueName: \"kubernetes.io/projected/5e8ba7d9-892e-4f45-846e-9d56500b25de-kube-api-access-hfwm6\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.390859 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.390788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1bf9f83-7660-43f9-8827-79237d176fcb-cert\") pod \"ingress-canary-b59sc\" (UID: \"c1bf9f83-7660-43f9-8827-79237d176fcb\") " pod="openshift-ingress-canary/ingress-canary-b59sc" Jul 28 10:08:10.390923 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.390860 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.390923 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.390904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8ba7d9-892e-4f45-846e-9d56500b25de-config-volume\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.391016 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.390931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8ba7d9-892e-4f45-846e-9d56500b25de-tmp-dir\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.391016 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.390958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxp7\" (UniqueName: \"kubernetes.io/projected/c1bf9f83-7660-43f9-8827-79237d176fcb-kube-api-access-tlxp7\") pod \"ingress-canary-b59sc\" (UID: \"c1bf9f83-7660-43f9-8827-79237d176fcb\") " pod="openshift-ingress-canary/ingress-canary-b59sc" Jul 28 10:08:10.391086 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.391041 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 10:08:10.391126 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.391108 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls podName:5e8ba7d9-892e-4f45-846e-9d56500b25de nodeName:}" failed. No retries permitted until 2026-07-28 10:08:10.89108623 +0000 UTC m=+35.333665186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls") pod "dns-default-gdk8g" (UID: "5e8ba7d9-892e-4f45-846e-9d56500b25de") : secret "dns-default-metrics-tls" not found Jul 28 10:08:10.391461 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.391434 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e8ba7d9-892e-4f45-846e-9d56500b25de-tmp-dir\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.391576 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.391556 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8ba7d9-892e-4f45-846e-9d56500b25de-config-volume\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.393884 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.393863 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1bf9f83-7660-43f9-8827-79237d176fcb-cert\") pod \"ingress-canary-b59sc\" (UID: \"c1bf9f83-7660-43f9-8827-79237d176fcb\") " pod="openshift-ingress-canary/ingress-canary-b59sc" Jul 28 10:08:10.402955 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.402936 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwm6\" (UniqueName: \"kubernetes.io/projected/5e8ba7d9-892e-4f45-846e-9d56500b25de-kube-api-access-hfwm6\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.403038 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.402976 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxp7\" (UniqueName: \"kubernetes.io/projected/c1bf9f83-7660-43f9-8827-79237d176fcb-kube-api-access-tlxp7\") pod \"ingress-canary-b59sc\" (UID: \"c1bf9f83-7660-43f9-8827-79237d176fcb\") " pod="openshift-ingress-canary/ingress-canary-b59sc" Jul 28 10:08:10.426199 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.426172 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b59sc" Jul 28 10:08:10.694052 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.693964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:10.694271 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.694129 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 10:08:10.694271 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.694151 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659cc6d558-mxqwp: secret "image-registry-tls" not found Jul 28 10:08:10.694271 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.694225 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls podName:03330411-ab4f-43a6-92f3-cefc7526cc9f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:11.694203382 +0000 UTC m=+36.136782335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls") pod "image-registry-659cc6d558-mxqwp" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f") : secret "image-registry-tls" not found Jul 28 10:08:10.795019 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.794979 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:10.795266 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.795066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.795266 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.795120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:10.795266 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.795145 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 10:08:10.795266 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.795216 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:11.795197643 +0000 UTC m=+36.237776602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : configmap references non-existent config key: service-ca.crt Jul 28 10:08:10.795266 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.795236 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls podName:17049657-4f76-49fa-b85c-d47d4bc68493 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:11.795227854 +0000 UTC m=+36.237806817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls") pod "cluster-samples-operator-6748489656-t54tx" (UID: "17049657-4f76-49fa-b85c-d47d4bc68493") : secret "samples-operator-tls" not found Jul 28 10:08:10.795266 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.795234 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:10.795573 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.795284 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 10:08:10.795573 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.795153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:10.795573 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.795289 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls podName:fcdc3068-bf6d-437d-9b29-1210b9eb6bd6 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:11.795274106 +0000 UTC m=+36.237853061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-s6m87" (UID: "fcdc3068-bf6d-437d-9b29-1210b9eb6bd6") : secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:10.795573 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.795353 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:11.795338842 +0000 UTC m=+36.237917801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : secret "router-metrics-certs-default" not found Jul 28 10:08:10.895970 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:10.895938 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:10.896185 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.896111 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 10:08:10.896243 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:10.896190 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls podName:5e8ba7d9-892e-4f45-846e-9d56500b25de nodeName:}" failed. No retries permitted until 2026-07-28 10:08:11.896169283 +0000 UTC m=+36.338748235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls") pod "dns-default-gdk8g" (UID: "5e8ba7d9-892e-4f45-846e-9d56500b25de") : secret "dns-default-metrics-tls" not found Jul 28 10:08:11.073413 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.073371 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:11.073413 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.073406 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:11.074251 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.073417 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:11.076336 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.076313 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 10:08:11.076468 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.076348 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 10:08:11.076510 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.076482 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rf2fq\"" Jul 28 10:08:11.076596 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.076579 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 10:08:11.076596 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.076591 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ltg76\"" Jul 28 10:08:11.076738 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.076648 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 10:08:11.583999 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.583966 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-wkm97"] Jul 28 10:08:11.585067 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.585035 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr"] Jul 28 10:08:11.586113 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.586080 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m"] Jul 28 10:08:11.588282 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:11.588253 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafae72a_446d_4852_9799_ed64ff1569fd.slice/crio-ed020bfa26918680914573e7601a0063727181d6963d5c0e5a5c324724d83435 WatchSource:0}: Error finding container ed020bfa26918680914573e7601a0063727181d6963d5c0e5a5c324724d83435: Status 404 returned error can't find the container with id ed020bfa26918680914573e7601a0063727181d6963d5c0e5a5c324724d83435 Jul 28 10:08:11.589380 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:11.589355 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977426ec_297c_41ec_ab46_c83d7cf4d480.slice/crio-8e0a78effe19a6e4934d70a39951449ec78a37b491bce3d34007f4624b66044a WatchSource:0}: Error finding container 8e0a78effe19a6e4934d70a39951449ec78a37b491bce3d34007f4624b66044a: Status 404 returned error can't find the container with id 8e0a78effe19a6e4934d70a39951449ec78a37b491bce3d34007f4624b66044a Jul 28 10:08:11.592240 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:11.590714 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25239465_8177_4ac4_9cb3_a293eb83589e.slice/crio-d5995bdf17e0d1e99fc88dc0cdb588e2bfe887b8eba49874ba08ce2f32b065db WatchSource:0}: Error finding container d5995bdf17e0d1e99fc88dc0cdb588e2bfe887b8eba49874ba08ce2f32b065db: Status 404 returned error can't find the container with id d5995bdf17e0d1e99fc88dc0cdb588e2bfe887b8eba49874ba08ce2f32b065db Jul 28 10:08:11.592563 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.592541 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw"] Jul 28 10:08:11.595422 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:11.595402 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056218d8_cf3f_4415_99c2_cac7b557f301.slice/crio-51c249c3a56f2706942f37e9a2354821a08aad33365dd746c4a2192f8e01e598 WatchSource:0}: Error finding container 51c249c3a56f2706942f37e9a2354821a08aad33365dd746c4a2192f8e01e598: Status 404 returned error can't find the container with id 51c249c3a56f2706942f37e9a2354821a08aad33365dd746c4a2192f8e01e598 Jul 28 10:08:11.598756 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.598736 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b59sc"] Jul 28 10:08:11.602274 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.602253 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-jd9hz"] Jul 28 10:08:11.605448 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:11.605416 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bf9f83_7660_43f9_8827_79237d176fcb.slice/crio-7cb36ef1d6aef564fc49ec6062ac58d4eca9d0dc5b288e3a2d86f17f23a700b2 WatchSource:0}: Error finding container 7cb36ef1d6aef564fc49ec6062ac58d4eca9d0dc5b288e3a2d86f17f23a700b2: Status 404 returned error can't find the container with id 7cb36ef1d6aef564fc49ec6062ac58d4eca9d0dc5b288e3a2d86f17f23a700b2 Jul 28 10:08:11.606986 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:11.606969 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba36116_0fd9_4d91_b070_c52a5378dc54.slice/crio-2b92ac2a4d31aab71e5192c4bc423e411b6150d69650f1f10b4b940d58973f87 WatchSource:0}: Error finding container 2b92ac2a4d31aab71e5192c4bc423e411b6150d69650f1f10b4b940d58973f87: Status 404 returned error can't find the container with id 2b92ac2a4d31aab71e5192c4bc423e411b6150d69650f1f10b4b940d58973f87 Jul 28 10:08:11.706741 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.706717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:11.706898 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.706880 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 10:08:11.706971 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.706902 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659cc6d558-mxqwp: secret "image-registry-tls" not found Jul 28 10:08:11.707022 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.706970 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls podName:03330411-ab4f-43a6-92f3-cefc7526cc9f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:13.706949165 +0000 UTC m=+38.149528119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls") pod "image-registry-659cc6d558-mxqwp" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f") : secret "image-registry-tls" not found Jul 28 10:08:11.807813 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.807785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:11.807923 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.807830 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:11.807923 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.807899 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:11.807993 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.807938 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 10:08:11.807993 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.807971 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 10:08:11.808050 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.807999 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:11.808050 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.808003 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:13.807984663 +0000 UTC m=+38.250563630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : secret "router-metrics-certs-default" not found Jul 28 10:08:11.808123 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.808071 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls podName:17049657-4f76-49fa-b85c-d47d4bc68493 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:13.808055571 +0000 UTC m=+38.250634524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls") pod "cluster-samples-operator-6748489656-t54tx" (UID: "17049657-4f76-49fa-b85c-d47d4bc68493") : secret "samples-operator-tls" not found Jul 28 10:08:11.808123 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.808088 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls podName:fcdc3068-bf6d-437d-9b29-1210b9eb6bd6 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:13.808081688 +0000 UTC m=+38.250660640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-s6m87" (UID: "fcdc3068-bf6d-437d-9b29-1210b9eb6bd6") : secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:11.808193 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.808145 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:11.808236 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.808227 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:13.80822024 +0000 UTC m=+38.250799193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : configmap references non-existent config key: service-ca.crt Jul 28 10:08:11.909001 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:11.908922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:11.909144 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.909042 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 10:08:11.909144 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:11.909094 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls podName:5e8ba7d9-892e-4f45-846e-9d56500b25de nodeName:}" failed. No retries permitted until 2026-07-28 10:08:13.909080373 +0000 UTC m=+38.351659326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls") pod "dns-default-gdk8g" (UID: "5e8ba7d9-892e-4f45-846e-9d56500b25de") : secret "dns-default-metrics-tls" not found Jul 28 10:08:12.334226 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:12.334190 2569 generic.go:356] "Generic (PLEG): container finished" podID="dea71310-9b24-40e8-8b58-8780a189b3df" containerID="6be36949f4cd82d4daf4aa8c392f4946b4a69b608ebbe4addf19ce1d02a6bc17" exitCode=0 Jul 28 10:08:12.334985 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:12.334285 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75jhn" event={"ID":"dea71310-9b24-40e8-8b58-8780a189b3df","Type":"ContainerDied","Data":"6be36949f4cd82d4daf4aa8c392f4946b4a69b608ebbe4addf19ce1d02a6bc17"} Jul 28 10:08:12.337010 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:12.336984 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" event={"ID":"25239465-8177-4ac4-9cb3-a293eb83589e","Type":"ContainerStarted","Data":"d5995bdf17e0d1e99fc88dc0cdb588e2bfe887b8eba49874ba08ce2f32b065db"} Jul 28 10:08:12.339081 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:12.339019 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw" event={"ID":"056218d8-cf3f-4415-99c2-cac7b557f301","Type":"ContainerStarted","Data":"51c249c3a56f2706942f37e9a2354821a08aad33365dd746c4a2192f8e01e598"} Jul 28 10:08:12.342190 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:12.342128 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" event={"ID":"fafae72a-446d-4852-9799-ed64ff1569fd","Type":"ContainerStarted","Data":"ed020bfa26918680914573e7601a0063727181d6963d5c0e5a5c324724d83435"} Jul 28 10:08:12.345836 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:12.345810 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b59sc" event={"ID":"c1bf9f83-7660-43f9-8827-79237d176fcb","Type":"ContainerStarted","Data":"7cb36ef1d6aef564fc49ec6062ac58d4eca9d0dc5b288e3a2d86f17f23a700b2"} Jul 28 10:08:12.348611 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:12.348554 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" event={"ID":"977426ec-297c-41ec-ab46-c83d7cf4d480","Type":"ContainerStarted","Data":"8e0a78effe19a6e4934d70a39951449ec78a37b491bce3d34007f4624b66044a"} Jul 28 10:08:12.351176 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:12.351126 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-jd9hz" event={"ID":"6ba36116-0fd9-4d91-b070-c52a5378dc54","Type":"ContainerStarted","Data":"2b92ac2a4d31aab71e5192c4bc423e411b6150d69650f1f10b4b940d58973f87"} Jul 28 10:08:13.364341 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:13.364309 2569 generic.go:356] "Generic (PLEG): container finished" podID="dea71310-9b24-40e8-8b58-8780a189b3df" containerID="2b206a4bfa4fa45b7ff5f47e924bd698c9b5949ee2f5b6fd72130f5778184eb5" exitCode=0 Jul 28 10:08:13.364737 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:13.364368 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75jhn" event={"ID":"dea71310-9b24-40e8-8b58-8780a189b3df","Type":"ContainerDied","Data":"2b206a4bfa4fa45b7ff5f47e924bd698c9b5949ee2f5b6fd72130f5778184eb5"} Jul 28 10:08:13.731585 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:13.731153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:13.731791 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.731507 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 10:08:13.731791 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.731610 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659cc6d558-mxqwp: secret "image-registry-tls" not found Jul 28 10:08:13.731791 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.731698 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls podName:03330411-ab4f-43a6-92f3-cefc7526cc9f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:17.731678483 +0000 UTC m=+42.174257457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls") pod "image-registry-659cc6d558-mxqwp" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f") : secret "image-registry-tls" not found Jul 28 10:08:13.832799 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:13.832762 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:13.833050 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.833028 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 10:08:13.833471 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.833119 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls podName:17049657-4f76-49fa-b85c-d47d4bc68493 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:17.833094993 +0000 UTC m=+42.275673962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls") pod "cluster-samples-operator-6748489656-t54tx" (UID: "17049657-4f76-49fa-b85c-d47d4bc68493") : secret "samples-operator-tls" not found Jul 28 10:08:13.833663 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:13.833613 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:13.833748 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:13.833730 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:13.833748 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.833741 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:17.833723075 +0000 UTC m=+42.276302032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : configmap references non-existent config key: service-ca.crt Jul 28 10:08:13.833872 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:13.833783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:13.833872 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.833806 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 10:08:13.833872 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.833843 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:17.833831413 +0000 UTC m=+42.276410369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : secret "router-metrics-certs-default" not found Jul 28 10:08:13.833872 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.833862 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:13.834052 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.833894 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls podName:fcdc3068-bf6d-437d-9b29-1210b9eb6bd6 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:17.83388422 +0000 UTC m=+42.276463178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-s6m87" (UID: "fcdc3068-bf6d-437d-9b29-1210b9eb6bd6") : secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:13.934854 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:13.934811 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:13.935044 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.935009 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 10:08:13.935191 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:13.935075 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls podName:5e8ba7d9-892e-4f45-846e-9d56500b25de nodeName:}" failed. No retries permitted until 2026-07-28 10:08:17.935055424 +0000 UTC m=+42.377634397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls") pod "dns-default-gdk8g" (UID: "5e8ba7d9-892e-4f45-846e-9d56500b25de") : secret "dns-default-metrics-tls" not found Jul 28 10:08:14.371219 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:14.371178 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-75jhn" event={"ID":"dea71310-9b24-40e8-8b58-8780a189b3df","Type":"ContainerStarted","Data":"7bd1f44ef12be6801ffae2f767a7c1ea8ae21d438393715bd72ff4bb99681a98"} Jul 28 10:08:14.372855 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:14.372819 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw" event={"ID":"056218d8-cf3f-4415-99c2-cac7b557f301","Type":"ContainerStarted","Data":"3989d1ff1167975a8c6a097466498303d6d2a9d30246718db2d8ebb81dcffdfe"} Jul 28 10:08:14.424634 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:14.424566 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-75jhn" podStartSLOduration=4.18330648 podStartE2EDuration="38.424551935s" totalImagesPullingTime="34.241245455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-28 10:08:14.422326752 +0000 UTC m=+38.864905727" watchObservedRunningTime="2026-07-28 10:08:14.424551935 +0000 UTC m=+38.867130910" Jul 28 10:08:14.441064 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:14.441016 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-npsmw" podStartSLOduration=8.680601169 podStartE2EDuration="10.440996792s" totalImagesPullingTime="1.760395623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:14.440157203 +0000 UTC m=+38.882736178" watchObservedRunningTime="2026-07-28 10:08:14.440996792 +0000 UTC m=+38.883575769" Jul 28 10:08:17.770045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:17.770007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:17.770470 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.770249 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 10:08:17.770470 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.770263 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659cc6d558-mxqwp: secret "image-registry-tls" not found Jul 28 10:08:17.770470 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.770315 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls podName:03330411-ab4f-43a6-92f3-cefc7526cc9f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:25.77029665 +0000 UTC m=+50.212875605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls") pod "image-registry-659cc6d558-mxqwp" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f") : secret "image-registry-tls" not found Jul 28 10:08:17.870615 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:17.870566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:17.870788 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.870740 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:17.870788 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:17.870759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:17.870913 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.870815 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls podName:fcdc3068-bf6d-437d-9b29-1210b9eb6bd6 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:25.87079516 +0000 UTC m=+50.313374128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-s6m87" (UID: "fcdc3068-bf6d-437d-9b29-1210b9eb6bd6") : secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:17.870913 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.870870 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 10:08:17.870913 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:17.870876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:17.871077 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.870922 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls podName:17049657-4f76-49fa-b85c-d47d4bc68493 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:25.870904387 +0000 UTC m=+50.313483346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls") pod "cluster-samples-operator-6748489656-t54tx" (UID: "17049657-4f76-49fa-b85c-d47d4bc68493") : secret "samples-operator-tls" not found Jul 28 10:08:17.871077 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.870964 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:25.870953525 +0000 UTC m=+50.313532484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : configmap references non-existent config key: service-ca.crt Jul 28 10:08:17.871077 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:17.870994 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:17.871244 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.871083 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 10:08:17.871244 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.871112 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:25.871101833 +0000 UTC m=+50.313680790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : secret "router-metrics-certs-default" not found Jul 28 10:08:17.972320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:17.972284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:17.972510 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.972443 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 10:08:17.972578 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:17.972517 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls podName:5e8ba7d9-892e-4f45-846e-9d56500b25de nodeName:}" failed. No retries permitted until 2026-07-28 10:08:25.972497612 +0000 UTC m=+50.415076585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls") pod "dns-default-gdk8g" (UID: "5e8ba7d9-892e-4f45-846e-9d56500b25de") : secret "dns-default-metrics-tls" not found Jul 28 10:08:18.386532 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.386480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" event={"ID":"25239465-8177-4ac4-9cb3-a293eb83589e","Type":"ContainerStarted","Data":"b3874147583246bd15f4fc5e7249b8d831dfeab84bf3a6561891b1d92b6a5892"} Jul 28 10:08:18.388358 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.388338 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/0.log" Jul 28 10:08:18.388482 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.388372 2569 generic.go:356] "Generic (PLEG): container finished" podID="fafae72a-446d-4852-9799-ed64ff1569fd" containerID="db46ef509f4a435ae4281ed631778881d1b95c7f505c8d7b12d36e53d5d7b18b" exitCode=255 Jul 28 10:08:18.388482 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.388440 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" event={"ID":"fafae72a-446d-4852-9799-ed64ff1569fd","Type":"ContainerDied","Data":"db46ef509f4a435ae4281ed631778881d1b95c7f505c8d7b12d36e53d5d7b18b"} Jul 28 10:08:18.388679 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.388660 2569 scope.go:117] "RemoveContainer" containerID="db46ef509f4a435ae4281ed631778881d1b95c7f505c8d7b12d36e53d5d7b18b" Jul 28 10:08:18.390562 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.390187 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b59sc" event={"ID":"c1bf9f83-7660-43f9-8827-79237d176fcb","Type":"ContainerStarted","Data":"f7a0647952082f4ec4aa3cb2278ab200186545f710fa6051780abfbf2bf5efe9"} Jul 28 10:08:18.391879 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.391849 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" event={"ID":"977426ec-297c-41ec-ab46-c83d7cf4d480","Type":"ContainerStarted","Data":"42cdadd63c0fd664aed8bd5e8f9b64a58aa383d5512a94319cddf30ae4b01ae7"} Jul 28 10:08:18.393370 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.393347 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-jd9hz" event={"ID":"6ba36116-0fd9-4d91-b070-c52a5378dc54","Type":"ContainerStarted","Data":"56d9b7f4e82730b01c2f8e1658ee35a8907a1549d0de1175eff7c1daf0723ed8"} Jul 28 10:08:18.407199 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.407161 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" podStartSLOduration=8.45967053 podStartE2EDuration="14.407144609s" totalImagesPullingTime="5.947474079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:18.40511161 +0000 UTC m=+42.847690586" watchObservedRunningTime="2026-07-28 10:08:18.407144609 +0000 UTC m=+42.849723584" Jul 28 10:08:18.449547 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.449503 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" podStartSLOduration=8.505049985 podStartE2EDuration="14.44948783s" totalImagesPullingTime="5.944437845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:18.447922786 +0000 UTC m=+42.890501762" watchObservedRunningTime="2026-07-28 10:08:18.44948783 +0000 UTC m=+42.892066806" Jul 28 10:08:18.474708 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.474664 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-jd9hz" podStartSLOduration=8.547762565 podStartE2EDuration="14.47464653s" totalImagesPullingTime="5.926883965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:18.474525741 +0000 UTC m=+42.917104717" watchObservedRunningTime="2026-07-28 10:08:18.47464653 +0000 UTC m=+42.917225506" Jul 28 10:08:18.501128 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:18.498990 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b59sc" podStartSLOduration=3.569761288 podStartE2EDuration="9.498974614s" totalImagesPullingTime="5.929213326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:18.497921526 +0000 UTC m=+42.940500523" watchObservedRunningTime="2026-07-28 10:08:18.498974614 +0000 UTC m=+42.941553620" Jul 28 10:08:19.399632 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.399601 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/1.log" Jul 28 10:08:19.400083 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.400059 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/0.log" Jul 28 10:08:19.400148 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.400107 2569 generic.go:356] "Generic (PLEG): container finished" podID="fafae72a-446d-4852-9799-ed64ff1569fd" containerID="dada713776f65d8a81eb06143e292a249d1a02380da7025331e26fe235695dd4" exitCode=255 Jul 28 10:08:19.400253 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.400220 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" event={"ID":"fafae72a-446d-4852-9799-ed64ff1569fd","Type":"ContainerDied","Data":"dada713776f65d8a81eb06143e292a249d1a02380da7025331e26fe235695dd4"} Jul 28 10:08:19.400381 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.400287 2569 scope.go:117] "RemoveContainer" containerID="db46ef509f4a435ae4281ed631778881d1b95c7f505c8d7b12d36e53d5d7b18b" Jul 28 10:08:19.400449 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.400431 2569 scope.go:117] "RemoveContainer" containerID="dada713776f65d8a81eb06143e292a249d1a02380da7025331e26fe235695dd4" Jul 28 10:08:19.400755 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:19.400612 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-wkm97_openshift-console-operator(fafae72a-446d-4852-9799-ed64ff1569fd)\"" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" podUID="fafae72a-446d-4852-9799-ed64ff1569fd" Jul 28 10:08:19.520016 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.519980 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q5dpw"] Jul 28 10:08:19.550864 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.550835 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q5dpw"] Jul 28 10:08:19.551008 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.550967 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.554213 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.554181 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 10:08:19.554896 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.554581 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 10:08:19.554896 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.554718 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n7kvl\"" Jul 28 10:08:19.586060 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.586034 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.586195 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.586089 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de963509-197c-4ed1-af8b-afecc19b465e-data-volume\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.586263 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.586214 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de963509-197c-4ed1-af8b-afecc19b465e-crio-socket\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.586263 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.586252 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6f78\" (UniqueName: \"kubernetes.io/projected/de963509-197c-4ed1-af8b-afecc19b465e-kube-api-access-s6f78\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.586411 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.586302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.687285 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.687211 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.687285 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.687261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de963509-197c-4ed1-af8b-afecc19b465e-data-volume\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.687450 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.687324 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de963509-197c-4ed1-af8b-afecc19b465e-crio-socket\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.687450 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:19.687331 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 10:08:19.687450 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.687350 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6f78\" (UniqueName: \"kubernetes.io/projected/de963509-197c-4ed1-af8b-afecc19b465e-kube-api-access-s6f78\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.687450 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:19.687397 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm podName:de963509-197c-4ed1-af8b-afecc19b465e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:20.187377551 +0000 UTC m=+44.629956526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5dpw" (UID: "de963509-197c-4ed1-af8b-afecc19b465e") : configmap "kube-rbac-proxy" not found Jul 28 10:08:19.687649 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.687502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.687649 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.687541 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/de963509-197c-4ed1-af8b-afecc19b465e-crio-socket\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.687757 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.687668 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/de963509-197c-4ed1-af8b-afecc19b465e-data-volume\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:19.687757 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:19.687704 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 10:08:19.687849 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:19.687777 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls podName:de963509-197c-4ed1-af8b-afecc19b465e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:20.187757128 +0000 UTC m=+44.630336087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5dpw" (UID: "de963509-197c-4ed1-af8b-afecc19b465e") : secret "insights-runtime-extractor-tls" not found Jul 28 10:08:19.708007 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:19.707976 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6f78\" (UniqueName: \"kubernetes.io/projected/de963509-197c-4ed1-af8b-afecc19b465e-kube-api-access-s6f78\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:20.191335 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:20.191299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:20.191511 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:20.191399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:20.191511 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:20.191419 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 10:08:20.191511 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:20.191482 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm podName:de963509-197c-4ed1-af8b-afecc19b465e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:21.191465287 +0000 UTC m=+45.634044265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5dpw" (UID: "de963509-197c-4ed1-af8b-afecc19b465e") : configmap "kube-rbac-proxy" not found Jul 28 10:08:20.191642 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:20.191542 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 10:08:20.191642 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:20.191595 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls podName:de963509-197c-4ed1-af8b-afecc19b465e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:21.191581946 +0000 UTC m=+45.634160899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5dpw" (UID: "de963509-197c-4ed1-af8b-afecc19b465e") : secret "insights-runtime-extractor-tls" not found Jul 28 10:08:20.380450 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:20.380416 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:20.380610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:20.380467 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:20.404760 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:20.404737 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/1.log" Jul 28 10:08:20.405118 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:20.405059 2569 scope.go:117] "RemoveContainer" containerID="dada713776f65d8a81eb06143e292a249d1a02380da7025331e26fe235695dd4" Jul 28 10:08:20.405218 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:20.405200 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-wkm97_openshift-console-operator(fafae72a-446d-4852-9799-ed64ff1569fd)\"" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" podUID="fafae72a-446d-4852-9799-ed64ff1569fd" Jul 28 10:08:21.198985 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:21.198944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:21.199325 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:21.199033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:21.199325 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:21.199088 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 10:08:21.199325 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:21.199154 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm podName:de963509-197c-4ed1-af8b-afecc19b465e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:23.199138441 +0000 UTC m=+47.641717393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5dpw" (UID: "de963509-197c-4ed1-af8b-afecc19b465e") : configmap "kube-rbac-proxy" not found Jul 28 10:08:21.199325 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:21.199180 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 10:08:21.199325 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:21.199232 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls podName:de963509-197c-4ed1-af8b-afecc19b465e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:23.199219771 +0000 UTC m=+47.641798724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5dpw" (UID: "de963509-197c-4ed1-af8b-afecc19b465e") : secret "insights-runtime-extractor-tls" not found Jul 28 10:08:21.407992 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:21.407968 2569 scope.go:117] "RemoveContainer" containerID="dada713776f65d8a81eb06143e292a249d1a02380da7025331e26fe235695dd4" Jul 28 10:08:21.408379 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:21.408125 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-wkm97_openshift-console-operator(fafae72a-446d-4852-9799-ed64ff1569fd)\"" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" podUID="fafae72a-446d-4852-9799-ed64ff1569fd" Jul 28 10:08:21.700734 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:21.700709 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tp929_022e6ace-b742-4eb3-a13a-97d1ec1f10e3/dns-node-resolver/0.log" Jul 28 10:08:22.104214 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:22.104187 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mj48w_1acd164c-032f-40eb-bef6-b07042848a90/node-ca/0.log" Jul 28 10:08:22.905690 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:22.905663 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b59sc_c1bf9f83-7660-43f9-8827-79237d176fcb/serve-healthcheck-canary/0.log" Jul 28 10:08:23.216435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:23.216359 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:23.216574 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:23.216463 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:23.216574 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:23.216507 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 10:08:23.216574 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:23.216565 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 10:08:23.216574 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:23.216572 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls podName:de963509-197c-4ed1-af8b-afecc19b465e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:27.216557077 +0000 UTC m=+51.659136029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5dpw" (UID: "de963509-197c-4ed1-af8b-afecc19b465e") : secret "insights-runtime-extractor-tls" not found Jul 28 10:08:23.216735 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:23.216606 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm podName:de963509-197c-4ed1-af8b-afecc19b465e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:27.216595182 +0000 UTC m=+51.659174136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5dpw" (UID: "de963509-197c-4ed1-af8b-afecc19b465e") : configmap "kube-rbac-proxy" not found Jul 28 10:08:25.842036 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:25.841991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:25.842418 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.842137 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 10:08:25.842418 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.842156 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659cc6d558-mxqwp: secret "image-registry-tls" not found Jul 28 10:08:25.842418 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.842220 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls podName:03330411-ab4f-43a6-92f3-cefc7526cc9f nodeName:}" failed. No retries permitted until 2026-07-28 10:08:41.842204083 +0000 UTC m=+66.284783036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls") pod "image-registry-659cc6d558-mxqwp" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f") : secret "image-registry-tls" not found Jul 28 10:08:25.942931 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:25.942896 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:25.943105 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:25.942963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:25.943105 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:25.943012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:25.943105 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:25.943035 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:25.943105 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.943064 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 10:08:25.943105 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.943086 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:41.943069842 +0000 UTC m=+66.385648795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : configmap references non-existent config key: service-ca.crt Jul 28 10:08:25.943287 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.943136 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls podName:17049657-4f76-49fa-b85c-d47d4bc68493 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:41.943119558 +0000 UTC m=+66.385698535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls") pod "cluster-samples-operator-6748489656-t54tx" (UID: "17049657-4f76-49fa-b85c-d47d4bc68493") : secret "samples-operator-tls" not found Jul 28 10:08:25.943287 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.943136 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:25.943287 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.943144 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 10:08:25.943287 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.943176 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls podName:fcdc3068-bf6d-437d-9b29-1210b9eb6bd6 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:41.94316581 +0000 UTC m=+66.385744766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-s6m87" (UID: "fcdc3068-bf6d-437d-9b29-1210b9eb6bd6") : secret "cluster-monitoring-operator-tls" not found Jul 28 10:08:25.943287 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:25.943190 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs podName:0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac nodeName:}" failed. No retries permitted until 2026-07-28 10:08:41.943183532 +0000 UTC m=+66.385762487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs") pod "router-default-6b5bf78f8f-wkjq5" (UID: "0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac") : secret "router-metrics-certs-default" not found Jul 28 10:08:26.043423 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:26.043393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:26.043611 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:26.043554 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 10:08:26.043703 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:26.043690 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls podName:5e8ba7d9-892e-4f45-846e-9d56500b25de nodeName:}" failed. No retries permitted until 2026-07-28 10:08:42.043667278 +0000 UTC m=+66.486246231 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls") pod "dns-default-gdk8g" (UID: "5e8ba7d9-892e-4f45-846e-9d56500b25de") : secret "dns-default-metrics-tls" not found Jul 28 10:08:27.252846 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:27.252808 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:27.253201 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:27.252908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:27.253201 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:27.252960 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 10:08:27.253201 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:27.253024 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls podName:de963509-197c-4ed1-af8b-afecc19b465e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:35.253007931 +0000 UTC m=+59.695586884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5dpw" (UID: "de963509-197c-4ed1-af8b-afecc19b465e") : secret "insights-runtime-extractor-tls" not found Jul 28 10:08:27.253440 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:27.253423 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/de963509-197c-4ed1-af8b-afecc19b465e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:35.073915 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.073887 2569 scope.go:117] "RemoveContainer" containerID="dada713776f65d8a81eb06143e292a249d1a02380da7025331e26fe235695dd4" Jul 28 10:08:35.320113 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.320082 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:35.322071 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.322048 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/de963509-197c-4ed1-af8b-afecc19b465e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5dpw\" (UID: \"de963509-197c-4ed1-af8b-afecc19b465e\") " pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:35.326348 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.326295 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wkd29" Jul 28 10:08:35.451820 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.451791 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:08:35.452220 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.452206 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/1.log" Jul 28 10:08:35.452288 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.452236 2569 generic.go:356] "Generic (PLEG): container finished" podID="fafae72a-446d-4852-9799-ed64ff1569fd" containerID="3f0d319e682fe32d40d0e713abe3afff7b57de79fd917fe4b9682aec9a13ad63" exitCode=255 Jul 28 10:08:35.452288 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.452261 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" event={"ID":"fafae72a-446d-4852-9799-ed64ff1569fd","Type":"ContainerDied","Data":"3f0d319e682fe32d40d0e713abe3afff7b57de79fd917fe4b9682aec9a13ad63"} Jul 28 10:08:35.452356 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.452328 2569 scope.go:117] "RemoveContainer" containerID="dada713776f65d8a81eb06143e292a249d1a02380da7025331e26fe235695dd4" Jul 28 10:08:35.452603 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.452582 2569 scope.go:117] "RemoveContainer" containerID="3f0d319e682fe32d40d0e713abe3afff7b57de79fd917fe4b9682aec9a13ad63" Jul 28 10:08:35.452785 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:35.452766 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-wkm97_openshift-console-operator(fafae72a-446d-4852-9799-ed64ff1569fd)\"" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" podUID="fafae72a-446d-4852-9799-ed64ff1569fd" Jul 28 10:08:35.462027 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.462012 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q5dpw" Jul 28 10:08:35.612082 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:35.612058 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q5dpw"] Jul 28 10:08:35.613445 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:35.613421 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde963509_197c_4ed1_af8b_afecc19b465e.slice/crio-f445b429c98f20d132555297bd7cbcd48229d25ced9215419e605fec11a77ee7 WatchSource:0}: Error finding container f445b429c98f20d132555297bd7cbcd48229d25ced9215419e605fec11a77ee7: Status 404 returned error can't find the container with id f445b429c98f20d132555297bd7cbcd48229d25ced9215419e605fec11a77ee7 Jul 28 10:08:36.460449 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:36.460417 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:08:36.463565 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:36.463528 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5dpw" event={"ID":"de963509-197c-4ed1-af8b-afecc19b465e","Type":"ContainerStarted","Data":"f445b429c98f20d132555297bd7cbcd48229d25ced9215419e605fec11a77ee7"} Jul 28 10:08:40.380726 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.380688 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:40.381231 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.380754 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:08:40.381231 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.381135 2569 scope.go:117] "RemoveContainer" containerID="3f0d319e682fe32d40d0e713abe3afff7b57de79fd917fe4b9682aec9a13ad63" Jul 28 10:08:40.381359 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:40.381332 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-wkm97_openshift-console-operator(fafae72a-446d-4852-9799-ed64ff1569fd)\"" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" podUID="fafae72a-446d-4852-9799-ed64ff1569fd" Jul 28 10:08:40.873161 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.873125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:40.873321 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.873239 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:40.876251 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.876233 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 10:08:40.876251 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.876248 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 10:08:40.885656 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.885614 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbac8868-5bb5-4857-895f-455556932955-original-pull-secret\") pod \"global-pull-secret-syncer-q6r8g\" (UID: \"cbac8868-5bb5-4857-895f-455556932955\") " pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:40.885742 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.885695 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06b28783-48ec-4b28-94de-385cd15baaa8-metrics-certs\") pod \"network-metrics-daemon-trrp4\" (UID: \"06b28783-48ec-4b28-94de-385cd15baaa8\") " pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:40.974439 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.974404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:40.977691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.977669 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 10:08:40.987901 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.987879 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 10:08:40.997775 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:40.997752 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdgc\" (UniqueName: \"kubernetes.io/projected/d6b7677d-6c6c-4be6-893c-936e9ca050be-kube-api-access-4vdgc\") pod \"network-check-target-fr92k\" (UID: \"d6b7677d-6c6c-4be6-893c-936e9ca050be\") " pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:41.089261 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.089198 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ltg76\"" Jul 28 10:08:41.096958 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.096413 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:41.096958 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.096727 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rf2fq\"" Jul 28 10:08:41.100034 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.099873 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6r8g" Jul 28 10:08:41.104112 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.104089 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-trrp4" Jul 28 10:08:41.282061 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.282033 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fr92k"] Jul 28 10:08:41.287375 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:41.287335 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b7677d_6c6c_4be6_893c_936e9ca050be.slice/crio-1aae9b2085e5faa08ea675fa3b28927f537bbee2305340819bb9812c10c7c855 WatchSource:0}: Error finding container 1aae9b2085e5faa08ea675fa3b28927f537bbee2305340819bb9812c10c7c855: Status 404 returned error can't find the container with id 1aae9b2085e5faa08ea675fa3b28927f537bbee2305340819bb9812c10c7c855 Jul 28 10:08:41.294645 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.294566 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q6r8g"] Jul 28 10:08:41.296056 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:41.296023 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbac8868_5bb5_4857_895f_455556932955.slice/crio-71d1c89fcf35ca51d2e86ee30e5b0bac4dfc55ab9fde3cf7892f9aff1a323bcb WatchSource:0}: Error finding container 71d1c89fcf35ca51d2e86ee30e5b0bac4dfc55ab9fde3cf7892f9aff1a323bcb: Status 404 returned error can't find the container with id 71d1c89fcf35ca51d2e86ee30e5b0bac4dfc55ab9fde3cf7892f9aff1a323bcb Jul 28 10:08:41.315186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.315157 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-trrp4"] Jul 28 10:08:41.320689 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:41.320657 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b28783_48ec_4b28_94de_385cd15baaa8.slice/crio-578133c70e2324d242eb907ca58d7f79b2a6ef51a400b3e2071fed7d689fea77 WatchSource:0}: Error finding container 578133c70e2324d242eb907ca58d7f79b2a6ef51a400b3e2071fed7d689fea77: Status 404 returned error can't find the container with id 578133c70e2324d242eb907ca58d7f79b2a6ef51a400b3e2071fed7d689fea77 Jul 28 10:08:41.483548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.483470 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5dpw" event={"ID":"de963509-197c-4ed1-af8b-afecc19b465e","Type":"ContainerStarted","Data":"abdee833c75b8b65bef1e8180a32aaeb8bdc988429136311a06f597c64cc8c0f"} Jul 28 10:08:41.485279 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.485240 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fr92k" event={"ID":"d6b7677d-6c6c-4be6-893c-936e9ca050be","Type":"ContainerStarted","Data":"1aae9b2085e5faa08ea675fa3b28927f537bbee2305340819bb9812c10c7c855"} Jul 28 10:08:41.486670 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.486586 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-trrp4" event={"ID":"06b28783-48ec-4b28-94de-385cd15baaa8","Type":"ContainerStarted","Data":"578133c70e2324d242eb907ca58d7f79b2a6ef51a400b3e2071fed7d689fea77"} Jul 28 10:08:41.488307 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.488282 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q6r8g" event={"ID":"cbac8868-5bb5-4857-895f-455556932955","Type":"ContainerStarted","Data":"71d1c89fcf35ca51d2e86ee30e5b0bac4dfc55ab9fde3cf7892f9aff1a323bcb"} Jul 28 10:08:41.884644 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.884562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:41.888298 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.888245 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"image-registry-659cc6d558-mxqwp\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:41.986634 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.985974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:41.986634 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.986049 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:41.986634 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.986084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:41.986634 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.986144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:41.987789 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.987759 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-service-ca-bundle\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:41.989949 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.989900 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac-metrics-certs\") pod \"router-default-6b5bf78f8f-wkjq5\" (UID: \"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac\") " pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:41.991710 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.991653 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17049657-4f76-49fa-b85c-d47d4bc68493-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-t54tx\" (UID: \"17049657-4f76-49fa-b85c-d47d4bc68493\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:41.993902 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:41.993878 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fcdc3068-bf6d-437d-9b29-1210b9eb6bd6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-s6m87\" (UID: \"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:42.087189 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.087157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:42.090186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.090154 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e8ba7d9-892e-4f45-846e-9d56500b25de-metrics-tls\") pod \"dns-default-gdk8g\" (UID: \"5e8ba7d9-892e-4f45-846e-9d56500b25de\") " pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:42.102335 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.102312 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8x85w\"" Jul 28 10:08:42.109900 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.109879 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:42.158709 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.158684 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rf979\"" Jul 28 10:08:42.167244 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.166169 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" Jul 28 10:08:42.196873 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.196848 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4zl7s\"" Jul 28 10:08:42.204292 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.204268 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:42.211670 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.211652 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ndrbx\"" Jul 28 10:08:42.219698 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.219670 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" Jul 28 10:08:42.221036 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.221016 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5fpr8\"" Jul 28 10:08:42.230332 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.229069 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:42.349596 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.349215 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659cc6d558-mxqwp"] Jul 28 10:08:42.425348 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.425269 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87"] Jul 28 10:08:42.463795 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.463758 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b5bf78f8f-wkjq5"] Jul 28 10:08:42.495834 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.495761 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx"] Jul 28 10:08:42.503290 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.503246 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" event={"ID":"03330411-ab4f-43a6-92f3-cefc7526cc9f","Type":"ContainerStarted","Data":"1444023de931d654565782f5489d58701dfeee29724cea6d10cf26dc10d0253a"} Jul 28 10:08:42.527487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:42.527454 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gdk8g"] Jul 28 10:08:43.508485 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:43.508411 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gdk8g" event={"ID":"5e8ba7d9-892e-4f45-846e-9d56500b25de","Type":"ContainerStarted","Data":"64c3981d83f2a86bd72b17e56f780c3ad2a56856780d0295a219869025d82959"} Jul 28 10:08:43.509994 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:43.509901 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" event={"ID":"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac","Type":"ContainerStarted","Data":"b3057361e9f8a4249b8c7a39eb88146b05f9fb912fa2cc22726705dafc96792e"} Jul 28 10:08:43.511148 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:43.511100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" event={"ID":"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6","Type":"ContainerStarted","Data":"cf549ef2146ad77d49b57e50620b81231e7b5cffd93f1c2121d9463409234b3e"} Jul 28 10:08:45.517870 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:45.517831 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" event={"ID":"17049657-4f76-49fa-b85c-d47d4bc68493","Type":"ContainerStarted","Data":"3ce7cea6d2e5aa63a3e392961d6572a3ba99222a90c62219d7e7bdc5576de169"} Jul 28 10:08:50.538411 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.538374 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-trrp4" event={"ID":"06b28783-48ec-4b28-94de-385cd15baaa8","Type":"ContainerStarted","Data":"6af5e7b5e7f6f61ed792e20ebe9b7f873f0b7dd0d4a9b9d0571103540eacd1bf"} Jul 28 10:08:50.538897 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.538439 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-trrp4" event={"ID":"06b28783-48ec-4b28-94de-385cd15baaa8","Type":"ContainerStarted","Data":"d51353b8251cd9a21a9d729e35aaeace7b755bcf915f6ea7b6a1dedafe370bfd"} Jul 28 10:08:50.540048 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.540017 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" event={"ID":"fcdc3068-bf6d-437d-9b29-1210b9eb6bd6","Type":"ContainerStarted","Data":"b2e9e3fe017303970806da741cbfe65d359958d317505c6c76d4749d03e837b5"} Jul 28 10:08:50.541448 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.541423 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q6r8g" event={"ID":"cbac8868-5bb5-4857-895f-455556932955","Type":"ContainerStarted","Data":"7c6b78fa7b521d6f354369348e6de49adee502252025192b7c7963ca677e7c17"} Jul 28 10:08:50.543295 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.543270 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gdk8g" event={"ID":"5e8ba7d9-892e-4f45-846e-9d56500b25de","Type":"ContainerStarted","Data":"9869a9406346527048f4383d42248447d62129be45f82324b46319fb2a224b40"} Jul 28 10:08:50.543384 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.543320 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gdk8g" Jul 28 10:08:50.543384 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.543345 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gdk8g" event={"ID":"5e8ba7d9-892e-4f45-846e-9d56500b25de","Type":"ContainerStarted","Data":"5c2b096053bb2348a168c790fbda1d8ca7278ecae43ec29d600174c9f7930e21"} Jul 28 10:08:50.545195 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.545173 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5dpw" event={"ID":"de963509-197c-4ed1-af8b-afecc19b465e","Type":"ContainerStarted","Data":"ef7972305f18e0b055d9f13867f78d57083a7c6f63641f865b95e510f474823d"} Jul 28 10:08:50.546668 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.546646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fr92k" event={"ID":"d6b7677d-6c6c-4be6-893c-936e9ca050be","Type":"ContainerStarted","Data":"d9f3b84f058cd9c7701763d9d2561144dd2632657d0b2ac0e8a3d79a54b31926"} Jul 28 10:08:50.546779 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.546754 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:08:50.548057 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.548038 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" event={"ID":"0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac","Type":"ContainerStarted","Data":"1f115252ea6fb798e6b9762f21ced41d52defb08c6cbb6e653e0819415c20a34"} Jul 28 10:08:50.549580 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.549543 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" event={"ID":"03330411-ab4f-43a6-92f3-cefc7526cc9f","Type":"ContainerStarted","Data":"9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e"} Jul 28 10:08:50.549745 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.549699 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:08:50.551341 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.551322 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" event={"ID":"17049657-4f76-49fa-b85c-d47d4bc68493","Type":"ContainerStarted","Data":"deb6380b8b9df16f676be2721ef65087c59767149871366e32ebafa7058775d4"} Jul 28 10:08:50.551426 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.551362 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" event={"ID":"17049657-4f76-49fa-b85c-d47d4bc68493","Type":"ContainerStarted","Data":"72041ad77455b07d896c2a73220d6d2bf8aa3d390405ab2428ac467950589003"} Jul 28 10:08:50.559400 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.559365 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-trrp4" podStartSLOduration=66.729154965 podStartE2EDuration="1m14.559354501s" totalImagesPullingTime="7.830199536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.559088917 +0000 UTC m=+75.001667891" watchObservedRunningTime="2026-07-28 10:08:50.559354501 +0000 UTC m=+75.001933499" Jul 28 10:08:50.588267 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.588216 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" podStartSLOduration=74.588204165 podStartE2EDuration="1m14.588204165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.587271471 +0000 UTC m=+75.029850449" watchObservedRunningTime="2026-07-28 10:08:50.588204165 +0000 UTC m=+75.030783140" Jul 28 10:08:50.609864 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.609768 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gdk8g" podStartSLOduration=34.328217447 podStartE2EDuration="41.609751011s" totalImagesPullingTime="7.281533564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.60784269 +0000 UTC m=+75.050421655" watchObservedRunningTime="2026-07-28 10:08:50.609751011 +0000 UTC m=+75.052329991" Jul 28 10:08:50.626156 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.626118 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q6r8g" podStartSLOduration=66.770831557 podStartE2EDuration="1m14.626107466s" totalImagesPullingTime="7.855275909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.626105178 +0000 UTC m=+75.068684153" watchObservedRunningTime="2026-07-28 10:08:50.626107466 +0000 UTC m=+75.068686444" Jul 28 10:08:50.648652 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.648585 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fr92k" podStartSLOduration=65.856924311 podStartE2EDuration="1m14.648568171s" totalImagesPullingTime="8.79164386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.648267053 +0000 UTC m=+75.090846029" watchObservedRunningTime="2026-07-28 10:08:50.648568171 +0000 UTC m=+75.091147147" Jul 28 10:08:50.677732 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.677683 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-s6m87" podStartSLOduration=39.397415854 podStartE2EDuration="46.677668704s" totalImagesPullingTime="7.28025285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.677369319 +0000 UTC m=+75.119948294" watchObservedRunningTime="2026-07-28 10:08:50.677668704 +0000 UTC m=+75.120247705" Jul 28 10:08:50.710122 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.710067 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-t54tx" podStartSLOduration=41.306403315 podStartE2EDuration="46.710049635s" totalImagesPullingTime="5.40364632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.709179983 +0000 UTC m=+75.151758957" watchObservedRunningTime="2026-07-28 10:08:50.710049635 +0000 UTC m=+75.152628611" Jul 28 10:08:50.744949 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.744923 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c"] Jul 28 10:08:50.745213 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.745178 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" podStartSLOduration=46.745163712 podStartE2EDuration="46.745163712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.742711949 +0000 UTC m=+75.185290925" watchObservedRunningTime="2026-07-28 10:08:50.745163712 +0000 UTC m=+75.187742686" Jul 28 10:08:50.747263 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.747250 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" Jul 28 10:08:50.752872 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.752853 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 28 10:08:50.752997 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.752979 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7mrmm\"" Jul 28 10:08:50.767931 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.767910 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c"] Jul 28 10:08:50.864922 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.864838 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0d88a30e-5847-48db-a733-fc0e437d7980-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-c5p5c\" (UID: \"0d88a30e-5847-48db-a733-fc0e437d7980\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" Jul 28 10:08:50.965739 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.965712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0d88a30e-5847-48db-a733-fc0e437d7980-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-c5p5c\" (UID: \"0d88a30e-5847-48db-a733-fc0e437d7980\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" Jul 28 10:08:50.968194 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:50.968161 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0d88a30e-5847-48db-a733-fc0e437d7980-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-c5p5c\" (UID: \"0d88a30e-5847-48db-a733-fc0e437d7980\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" Jul 28 10:08:51.056240 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:51.056213 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" Jul 28 10:08:51.073856 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:51.073836 2569 scope.go:117] "RemoveContainer" containerID="3f0d319e682fe32d40d0e713abe3afff7b57de79fd917fe4b9682aec9a13ad63" Jul 28 10:08:51.074043 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:51.074023 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-wkm97_openshift-console-operator(fafae72a-446d-4852-9799-ed64ff1569fd)\"" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" podUID="fafae72a-446d-4852-9799-ed64ff1569fd" Jul 28 10:08:51.198343 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:51.198311 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c"] Jul 28 10:08:51.202796 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:51.202766 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d88a30e_5847_48db_a733_fc0e437d7980.slice/crio-d44d113a29df6ca294fe3d86d840a5e7e28d2dcbf1ac32336291fe75b66056c1 WatchSource:0}: Error finding container d44d113a29df6ca294fe3d86d840a5e7e28d2dcbf1ac32336291fe75b66056c1: Status 404 returned error can't find the container with id d44d113a29df6ca294fe3d86d840a5e7e28d2dcbf1ac32336291fe75b66056c1 Jul 28 10:08:51.204689 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:51.204637 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:51.207304 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:51.207282 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:51.557952 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:51.557910 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" event={"ID":"0d88a30e-5847-48db-a733-fc0e437d7980","Type":"ContainerStarted","Data":"d44d113a29df6ca294fe3d86d840a5e7e28d2dcbf1ac32336291fe75b66056c1"} Jul 28 10:08:51.558562 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:51.558536 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:51.559912 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:51.559888 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6b5bf78f8f-wkjq5" Jul 28 10:08:52.566354 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.566310 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" event={"ID":"0d88a30e-5847-48db-a733-fc0e437d7980","Type":"ContainerStarted","Data":"c1bb65497c73e40b28cdae426c9f1376f1d2f60f80212d7af7d12d7d12847441"} Jul 28 10:08:52.566819 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.566459 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" Jul 28 10:08:52.572181 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.572157 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" Jul 28 10:08:52.584417 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.584372 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-c5p5c" podStartSLOduration=1.419333409 podStartE2EDuration="2.584356209s" totalImagesPullingTime="1.1650228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:52.583959263 +0000 UTC m=+77.026538237" watchObservedRunningTime="2026-07-28 10:08:52.584356209 +0000 UTC m=+77.026935184" Jul 28 10:08:52.784943 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.784907 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2cvht"] Jul 28 10:08:52.788903 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.788881 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.792035 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.791992 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 10:08:52.792129 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.792080 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 10:08:52.792321 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.792297 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7ppjp\"" Jul 28 10:08:52.792321 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.792320 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 10:08:52.796498 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.796428 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2cvht"] Jul 28 10:08:52.883964 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.883928 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c240e311-cb8d-4abf-b063-e0dba541ebcc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.884165 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.883977 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c240e311-cb8d-4abf-b063-e0dba541ebcc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.884165 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.884006 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c240e311-cb8d-4abf-b063-e0dba541ebcc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.884165 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.884076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8xq\" (UniqueName: \"kubernetes.io/projected/c240e311-cb8d-4abf-b063-e0dba541ebcc-kube-api-access-sp8xq\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.985207 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.985175 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c240e311-cb8d-4abf-b063-e0dba541ebcc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.985401 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.985217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c240e311-cb8d-4abf-b063-e0dba541ebcc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.985401 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.985270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c240e311-cb8d-4abf-b063-e0dba541ebcc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.985401 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.985296 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8xq\" (UniqueName: \"kubernetes.io/projected/c240e311-cb8d-4abf-b063-e0dba541ebcc-kube-api-access-sp8xq\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.985794 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:52.985771 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 28 10:08:52.985862 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:08:52.985853 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c240e311-cb8d-4abf-b063-e0dba541ebcc-prometheus-operator-tls podName:c240e311-cb8d-4abf-b063-e0dba541ebcc nodeName:}" failed. No retries permitted until 2026-07-28 10:08:53.48583258 +0000 UTC m=+77.928411554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c240e311-cb8d-4abf-b063-e0dba541ebcc-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-2cvht" (UID: "c240e311-cb8d-4abf-b063-e0dba541ebcc") : secret "prometheus-operator-tls" not found Jul 28 10:08:52.986048 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.986013 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c240e311-cb8d-4abf-b063-e0dba541ebcc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.988050 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.988023 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c240e311-cb8d-4abf-b063-e0dba541ebcc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:52.995638 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:52.995601 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8xq\" (UniqueName: \"kubernetes.io/projected/c240e311-cb8d-4abf-b063-e0dba541ebcc-kube-api-access-sp8xq\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:53.488941 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:53.488906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c240e311-cb8d-4abf-b063-e0dba541ebcc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:53.491504 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:53.491474 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c240e311-cb8d-4abf-b063-e0dba541ebcc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2cvht\" (UID: \"c240e311-cb8d-4abf-b063-e0dba541ebcc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:53.700412 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:53.700377 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" Jul 28 10:08:54.217219 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:54.217196 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2cvht"] Jul 28 10:08:54.218936 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:54.218909 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc240e311_cb8d_4abf_b063_e0dba541ebcc.slice/crio-34b2b4acddec2ff91985cfa9ce0f4d9f50de951eb46b3d3187a53ae26f524386 WatchSource:0}: Error finding container 34b2b4acddec2ff91985cfa9ce0f4d9f50de951eb46b3d3187a53ae26f524386: Status 404 returned error can't find the container with id 34b2b4acddec2ff91985cfa9ce0f4d9f50de951eb46b3d3187a53ae26f524386 Jul 28 10:08:54.576048 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:54.576011 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5dpw" event={"ID":"de963509-197c-4ed1-af8b-afecc19b465e","Type":"ContainerStarted","Data":"41ec882e11a3ce153df3a5578bcaa907b7ed8f90ad03d3a452bba85bfb28dcaa"} Jul 28 10:08:54.577100 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:54.577076 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" event={"ID":"c240e311-cb8d-4abf-b063-e0dba541ebcc","Type":"ContainerStarted","Data":"34b2b4acddec2ff91985cfa9ce0f4d9f50de951eb46b3d3187a53ae26f524386"} Jul 28 10:08:54.597825 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:54.597787 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q5dpw" podStartSLOduration=17.2677778 podStartE2EDuration="35.597773172s" totalImagesPullingTime="18.329995372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:54.596237658 +0000 UTC m=+79.038816646" watchObservedRunningTime="2026-07-28 10:08:54.597773172 +0000 UTC m=+79.040352147" Jul 28 10:08:56.585864 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:56.585831 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" event={"ID":"c240e311-cb8d-4abf-b063-e0dba541ebcc","Type":"ContainerStarted","Data":"f6f1c3e0ebb1040795c4031de99391853ef82ee88d4d247937f077f527b53199"} Jul 28 10:08:56.586224 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:56.585879 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" event={"ID":"c240e311-cb8d-4abf-b063-e0dba541ebcc","Type":"ContainerStarted","Data":"40ac5b1ab6cf59fcc75cdf84cc734f134d9a508a97b42a47959bdf983fc75314"} Jul 28 10:08:56.609853 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:56.609806 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-2cvht" podStartSLOduration=3.256441608 podStartE2EDuration="4.609790437s" totalImagesPullingTime="1.353348829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:56.609253799 +0000 UTC m=+81.051832775" watchObservedRunningTime="2026-07-28 10:08:56.609790437 +0000 UTC m=+81.052369412" Jul 28 10:08:58.285886 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.285770 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xwvhp"] Jul 28 10:08:58.288953 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.288931 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.291988 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.291965 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 10:08:58.292457 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.292439 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-79tkk\"" Jul 28 10:08:58.292551 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.292485 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 10:08:58.293331 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.293314 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 10:08:58.426915 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.426880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-wtmp\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.426915 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.426915 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-sys\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.427103 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.426937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-accelerators-collector-config\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.427103 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.426958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-textfile\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.427103 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.427009 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.427103 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.427055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-root\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.427103 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.427078 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-tls\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.427103 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.427103 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqz6\" (UniqueName: \"kubernetes.io/projected/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-kube-api-access-vsqz6\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.427281 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.427142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-metrics-client-ca\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528253 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-textfile\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528253 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528455 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528274 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-root\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528455 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528288 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-tls\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528455 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528309 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqz6\" (UniqueName: \"kubernetes.io/projected/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-kube-api-access-vsqz6\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528455 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-metrics-client-ca\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528455 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528365 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-root\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528455 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528392 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-wtmp\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528455 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528428 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-sys\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528455 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-accelerators-collector-config\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528809 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528574 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-wtmp\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528809 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528665 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-sys\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.528809 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528668 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-textfile\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.529009 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.528985 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-metrics-client-ca\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.529068 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.529031 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-accelerators-collector-config\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.530660 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.530638 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.530768 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.530753 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-node-exporter-tls\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.539768 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.539717 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqz6\" (UniqueName: \"kubernetes.io/projected/e94544ef-0ac0-4cb8-a3ca-ba433190f28a-kube-api-access-vsqz6\") pod \"node-exporter-xwvhp\" (UID: \"e94544ef-0ac0-4cb8-a3ca-ba433190f28a\") " pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.598102 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:58.598070 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xwvhp" Jul 28 10:08:58.608950 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:08:58.608927 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94544ef_0ac0_4cb8_a3ca_ba433190f28a.slice/crio-e0c569d4810b32441ae922e97a9a187cc0c34d60b799042da6de691defe6fc72 WatchSource:0}: Error finding container e0c569d4810b32441ae922e97a9a187cc0c34d60b799042da6de691defe6fc72: Status 404 returned error can't find the container with id e0c569d4810b32441ae922e97a9a187cc0c34d60b799042da6de691defe6fc72 Jul 28 10:08:59.601492 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:08:59.601422 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xwvhp" event={"ID":"e94544ef-0ac0-4cb8-a3ca-ba433190f28a","Type":"ContainerStarted","Data":"e0c569d4810b32441ae922e97a9a187cc0c34d60b799042da6de691defe6fc72"} Jul 28 10:09:00.384090 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.384061 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-758cc57597-ktmkw"] Jul 28 10:09:00.386688 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.386670 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.390982 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.390960 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 10:09:00.391325 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.391308 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 10:09:00.391370 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.391351 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 10:09:00.392366 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.392343 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4quuq7g85dfe4\"" Jul 28 10:09:00.392479 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.392374 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 10:09:00.392585 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.392571 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5rd55\"" Jul 28 10:09:00.393454 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.393435 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 10:09:00.423098 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.423070 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-758cc57597-ktmkw"] Jul 28 10:09:00.543234 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.543201 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87951c92-a735-4cd3-b0ec-b486b04135c8-metrics-client-ca\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.543411 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.543239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-grpc-tls\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.543411 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.543270 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.543411 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.543359 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-tls\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.543579 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.543416 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.543579 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.543455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g46c\" (UniqueName: \"kubernetes.io/projected/87951c92-a735-4cd3-b0ec-b486b04135c8-kube-api-access-8g46c\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.543579 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.543492 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.543579 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.543532 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.560404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.560384 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gdk8g" Jul 28 10:09:00.608330 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.608284 2569 generic.go:356] "Generic (PLEG): container finished" podID="e94544ef-0ac0-4cb8-a3ca-ba433190f28a" containerID="fe1d49b6a40c7c7a91902ff0eb4870baf2c0e850275a89c98c525a246b62d3e0" exitCode=0 Jul 28 10:09:00.608820 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.608797 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xwvhp" event={"ID":"e94544ef-0ac0-4cb8-a3ca-ba433190f28a","Type":"ContainerDied","Data":"fe1d49b6a40c7c7a91902ff0eb4870baf2c0e850275a89c98c525a246b62d3e0"} Jul 28 10:09:00.644857 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.644798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87951c92-a735-4cd3-b0ec-b486b04135c8-metrics-client-ca\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.644857 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.644834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-grpc-tls\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.644857 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.644853 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.645007 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.644985 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-tls\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.645047 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.645029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.645106 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.645049 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g46c\" (UniqueName: \"kubernetes.io/projected/87951c92-a735-4cd3-b0ec-b486b04135c8-kube-api-access-8g46c\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.645106 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.645082 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.645201 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.645118 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.645657 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.645606 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/87951c92-a735-4cd3-b0ec-b486b04135c8-metrics-client-ca\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.647776 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.647749 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.647878 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.647792 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-grpc-tls\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.648027 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.648005 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.648149 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.648127 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.648286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.648270 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.648344 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.648329 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/87951c92-a735-4cd3-b0ec-b486b04135c8-secret-thanos-querier-tls\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.655919 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.655902 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g46c\" (UniqueName: \"kubernetes.io/projected/87951c92-a735-4cd3-b0ec-b486b04135c8-kube-api-access-8g46c\") pod \"thanos-querier-758cc57597-ktmkw\" (UID: \"87951c92-a735-4cd3-b0ec-b486b04135c8\") " pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.696146 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.696125 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:00.868418 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:00.868382 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-758cc57597-ktmkw"] Jul 28 10:09:00.873099 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:09:00.873073 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87951c92_a735_4cd3_b0ec_b486b04135c8.slice/crio-df061105b2b24a18d8d48bc3a5a6404c445f4684ede98e512f0b40213894eb20 WatchSource:0}: Error finding container df061105b2b24a18d8d48bc3a5a6404c445f4684ede98e512f0b40213894eb20: Status 404 returned error can't find the container with id df061105b2b24a18d8d48bc3a5a6404c445f4684ede98e512f0b40213894eb20 Jul 28 10:09:01.614707 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:01.614672 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xwvhp" event={"ID":"e94544ef-0ac0-4cb8-a3ca-ba433190f28a","Type":"ContainerStarted","Data":"6768094e98652388993759bba4eeca9cb580381bb7ec498a5ee571811a3d6512"} Jul 28 10:09:01.615177 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:01.614731 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xwvhp" event={"ID":"e94544ef-0ac0-4cb8-a3ca-ba433190f28a","Type":"ContainerStarted","Data":"334cf1c295859829e4348b2e5d557afdb2a970719dc93be60d09227977b60a78"} Jul 28 10:09:01.615977 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:01.615956 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" event={"ID":"87951c92-a735-4cd3-b0ec-b486b04135c8","Type":"ContainerStarted","Data":"df061105b2b24a18d8d48bc3a5a6404c445f4684ede98e512f0b40213894eb20"} Jul 28 10:09:01.639414 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:01.639372 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xwvhp" podStartSLOduration=2.633025644 podStartE2EDuration="3.639357442s" totalImagesPullingTime="941.409342ms" totalInitContainerRuntime="64.922456ms" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:01.63796243 +0000 UTC m=+86.080541406" watchObservedRunningTime="2026-07-28 10:09:01.639357442 +0000 UTC m=+86.081936410" Jul 28 10:09:03.626448 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:03.626411 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" event={"ID":"87951c92-a735-4cd3-b0ec-b486b04135c8","Type":"ContainerStarted","Data":"a8ae1041ef805b9c4178fc6ed4aee7ba994777f080c86a38949958b450ce0a6e"} Jul 28 10:09:03.626857 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:03.626481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" event={"ID":"87951c92-a735-4cd3-b0ec-b486b04135c8","Type":"ContainerStarted","Data":"4114cbffa3d1fe18931be3861598b8123e7a32c393f59ac88c2f605b9d3e8317"} Jul 28 10:09:03.626857 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:03.626506 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" event={"ID":"87951c92-a735-4cd3-b0ec-b486b04135c8","Type":"ContainerStarted","Data":"7838cfef8fccd9abd326f2cf30b8b5b9ff254b4017209cda887277497acbf289"} Jul 28 10:09:04.633509 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:04.633477 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" event={"ID":"87951c92-a735-4cd3-b0ec-b486b04135c8","Type":"ContainerStarted","Data":"b3ca52e02811f22bddce5bdf0fd011f7483b72e115199b83b551babda6888ebc"} Jul 28 10:09:04.633965 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:04.633547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" event={"ID":"87951c92-a735-4cd3-b0ec-b486b04135c8","Type":"ContainerStarted","Data":"ad22deb14df6f92bd7376580215d75d74919ee1731865a30534d5d8b5597f3d3"} Jul 28 10:09:04.633965 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:04.633579 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" event={"ID":"87951c92-a735-4cd3-b0ec-b486b04135c8","Type":"ContainerStarted","Data":"e5a3b9e504d8b7e6656025461266330cadacefb578ad7f9cfa8125fc53d5b010"} Jul 28 10:09:04.633965 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:04.633608 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:04.696696 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:04.696637 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" podStartSLOduration=2.113670977 podStartE2EDuration="4.69659935s" totalImagesPullingTime="2.582928373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:04.695933032 +0000 UTC m=+89.138512034" watchObservedRunningTime="2026-07-28 10:09:04.69659935 +0000 UTC m=+89.139178326" Jul 28 10:09:06.077073 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.077047 2569 scope.go:117] "RemoveContainer" containerID="3f0d319e682fe32d40d0e713abe3afff7b57de79fd917fe4b9682aec9a13ad63" Jul 28 10:09:06.643355 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.643322 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:09:06.643508 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.643444 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" event={"ID":"fafae72a-446d-4852-9799-ed64ff1569fd","Type":"ContainerStarted","Data":"d5efb90d5f1dd4801e63e87876ad2ac5b532faac2629dcb2bf86884355d730c1"} Jul 28 10:09:06.643761 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.643735 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:09:06.649321 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.649300 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" Jul 28 10:09:06.682636 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.682571 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-wkm97" podStartSLOduration=56.737292231 podStartE2EDuration="1m2.68254776s" totalImagesPullingTime="5.945255529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:06.682438237 +0000 UTC m=+91.125017212" watchObservedRunningTime="2026-07-28 10:09:06.68254776 +0000 UTC m=+91.125126734" Jul 28 10:09:06.734006 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.733975 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-2wp29"] Jul 28 10:09:06.736116 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.736095 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-2wp29" Jul 28 10:09:06.743402 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.743382 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-w4m4w\"" Jul 28 10:09:06.743571 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.743406 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 10:09:06.746196 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.746176 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 10:09:06.758177 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.758154 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-2wp29"] Jul 28 10:09:06.795030 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.795007 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tml6l\" (UniqueName: \"kubernetes.io/projected/9669856d-bc18-4b5c-8e4f-1398d95653ab-kube-api-access-tml6l\") pod \"downloads-7c97dc4555-2wp29\" (UID: \"9669856d-bc18-4b5c-8e4f-1398d95653ab\") " pod="openshift-console/downloads-7c97dc4555-2wp29" Jul 28 10:09:06.896198 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.896113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tml6l\" (UniqueName: \"kubernetes.io/projected/9669856d-bc18-4b5c-8e4f-1398d95653ab-kube-api-access-tml6l\") pod \"downloads-7c97dc4555-2wp29\" (UID: \"9669856d-bc18-4b5c-8e4f-1398d95653ab\") " pod="openshift-console/downloads-7c97dc4555-2wp29" Jul 28 10:09:06.905677 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:06.905637 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tml6l\" (UniqueName: \"kubernetes.io/projected/9669856d-bc18-4b5c-8e4f-1398d95653ab-kube-api-access-tml6l\") pod \"downloads-7c97dc4555-2wp29\" (UID: \"9669856d-bc18-4b5c-8e4f-1398d95653ab\") " pod="openshift-console/downloads-7c97dc4555-2wp29" Jul 28 10:09:07.045963 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:07.045933 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-2wp29" Jul 28 10:09:07.186378 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:07.186278 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-2wp29"] Jul 28 10:09:07.188926 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:09:07.188902 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9669856d_bc18_4b5c_8e4f_1398d95653ab.slice/crio-c4d6d2e46c50391919b2e34a047c2e66619b23b4a2e1eb3f1df9244987cd5754 WatchSource:0}: Error finding container c4d6d2e46c50391919b2e34a047c2e66619b23b4a2e1eb3f1df9244987cd5754: Status 404 returned error can't find the container with id c4d6d2e46c50391919b2e34a047c2e66619b23b4a2e1eb3f1df9244987cd5754 Jul 28 10:09:07.650463 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:07.650425 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-2wp29" event={"ID":"9669856d-bc18-4b5c-8e4f-1398d95653ab","Type":"ContainerStarted","Data":"c4d6d2e46c50391919b2e34a047c2e66619b23b4a2e1eb3f1df9244987cd5754"} Jul 28 10:09:10.643942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:10.643911 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-758cc57597-ktmkw" Jul 28 10:09:11.564265 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:11.564236 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:09:11.801765 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:11.801730 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-659cc6d558-mxqwp"] Jul 28 10:09:12.217188 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.217155 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64fb584c59-7lh2q"] Jul 28 10:09:12.236289 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.236261 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64fb584c59-7lh2q"] Jul 28 10:09:12.236465 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.236376 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.239493 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.239467 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 10:09:12.239697 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.239673 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 10:09:12.239790 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.239749 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7xhvl\"" Jul 28 10:09:12.239926 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.239912 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 10:09:12.240065 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.240039 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 10:09:12.240188 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.240166 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 10:09:12.344484 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.344449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-oauth-serving-cert\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.344675 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.344522 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-service-ca\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.344675 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.344554 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-oauth-config\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.344675 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.344648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-serving-cert\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.344810 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.344677 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-console-config\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.344810 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.344755 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqsp\" (UniqueName: \"kubernetes.io/projected/07155ef5-c784-42b7-be32-d8564c2afacd-kube-api-access-bjqsp\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.446065 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.446024 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqsp\" (UniqueName: \"kubernetes.io/projected/07155ef5-c784-42b7-be32-d8564c2afacd-kube-api-access-bjqsp\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.446264 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.446100 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-oauth-serving-cert\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.446264 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.446155 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-service-ca\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.446264 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.446185 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-oauth-config\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.446264 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.446220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-serving-cert\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.446264 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.446243 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-console-config\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.447010 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.446978 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-oauth-serving-cert\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.447010 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.446993 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-console-config\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.447453 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.447433 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-service-ca\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.449372 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.449346 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-oauth-config\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.449483 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.449375 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-serving-cert\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.459102 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.459082 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqsp\" (UniqueName: \"kubernetes.io/projected/07155ef5-c784-42b7-be32-d8564c2afacd-kube-api-access-bjqsp\") pod \"console-64fb584c59-7lh2q\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.549035 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.548994 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:12.697235 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:12.697209 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64fb584c59-7lh2q"] Jul 28 10:09:12.700260 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:09:12.700218 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07155ef5_c784_42b7_be32_d8564c2afacd.slice/crio-b3d344c083be32463bead1071c228b0b3bba9aaabec884b8818f9083b90b78e8 WatchSource:0}: Error finding container b3d344c083be32463bead1071c228b0b3bba9aaabec884b8818f9083b90b78e8: Status 404 returned error can't find the container with id b3d344c083be32463bead1071c228b0b3bba9aaabec884b8818f9083b90b78e8 Jul 28 10:09:13.683785 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:13.683742 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64fb584c59-7lh2q" event={"ID":"07155ef5-c784-42b7-be32-d8564c2afacd","Type":"ContainerStarted","Data":"b3d344c083be32463bead1071c228b0b3bba9aaabec884b8818f9083b90b78e8"} Jul 28 10:09:16.699123 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:16.699085 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64fb584c59-7lh2q" event={"ID":"07155ef5-c784-42b7-be32-d8564c2afacd","Type":"ContainerStarted","Data":"12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467"} Jul 28 10:09:16.722695 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:16.722647 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64fb584c59-7lh2q" podStartSLOduration=1.586362234 podStartE2EDuration="4.722631935s" totalImagesPullingTime="3.136269701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:16.721134692 +0000 UTC m=+101.163713668" watchObservedRunningTime="2026-07-28 10:09:16.722631935 +0000 UTC m=+101.165210902" Jul 28 10:09:19.213529 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.213490 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8fb97b8f7-ppntn"] Jul 28 10:09:19.326072 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.326039 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fb97b8f7-ppntn"] Jul 28 10:09:19.326259 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.326155 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.335432 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.335401 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 10:09:19.414267 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.414229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-service-ca\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.414437 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.414281 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-oauth-config\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.414437 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.414336 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-serving-cert\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.414437 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.414368 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rp8\" (UniqueName: \"kubernetes.io/projected/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-kube-api-access-75rp8\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.414437 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.414390 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-trusted-ca-bundle\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.414437 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.414409 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-oauth-serving-cert\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.414649 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.414473 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-config\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.515231 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.515197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-config\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.515407 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.515257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-service-ca\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.515407 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.515294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-oauth-config\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.515407 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.515321 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-serving-cert\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.515407 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.515351 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rp8\" (UniqueName: \"kubernetes.io/projected/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-kube-api-access-75rp8\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.515407 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.515376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-trusted-ca-bundle\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.515407 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.515407 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-oauth-serving-cert\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.516055 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.516005 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-config\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.516055 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.516010 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-oauth-serving-cert\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.516524 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.516491 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-trusted-ca-bundle\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.516691 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.516668 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-service-ca\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.518302 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.518281 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-oauth-config\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.518388 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.518316 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-serving-cert\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.524753 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.524702 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rp8\" (UniqueName: \"kubernetes.io/projected/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-kube-api-access-75rp8\") pod \"console-8fb97b8f7-ppntn\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:19.636708 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:19.636670 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:21.562491 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:21.562449 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fr92k" Jul 28 10:09:22.549877 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:22.549843 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:22.550175 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:22.549927 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:22.556562 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:22.556539 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:22.728572 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:22.728543 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:23.670981 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:23.670943 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fb97b8f7-ppntn"] Jul 28 10:09:23.672951 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:09:23.672922 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f9bf1b_bff4_4253_a16b_1d7a7ccf5412.slice/crio-a752c4ba0ec66421b15200635e466967b90ed3132aae695d41b30778f0b7a39f WatchSource:0}: Error finding container a752c4ba0ec66421b15200635e466967b90ed3132aae695d41b30778f0b7a39f: Status 404 returned error can't find the container with id a752c4ba0ec66421b15200635e466967b90ed3132aae695d41b30778f0b7a39f Jul 28 10:09:23.728440 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:23.728415 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fb97b8f7-ppntn" event={"ID":"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412","Type":"ContainerStarted","Data":"a752c4ba0ec66421b15200635e466967b90ed3132aae695d41b30778f0b7a39f"} Jul 28 10:09:24.734924 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:24.734877 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fb97b8f7-ppntn" event={"ID":"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412","Type":"ContainerStarted","Data":"fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582"} Jul 28 10:09:24.736758 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:24.736729 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-2wp29" event={"ID":"9669856d-bc18-4b5c-8e4f-1398d95653ab","Type":"ContainerStarted","Data":"88ac1bf687cd496ccc06892d88b2cefe501f0e7b99c6251ba40d973a7609f42a"} Jul 28 10:09:24.737054 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:24.737008 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-2wp29" Jul 28 10:09:24.751053 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:24.751031 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-2wp29" Jul 28 10:09:24.759888 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:24.759842 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8fb97b8f7-ppntn" podStartSLOduration=5.759827659 podStartE2EDuration="5.759827659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:24.75848232 +0000 UTC m=+109.201061324" watchObservedRunningTime="2026-07-28 10:09:24.759827659 +0000 UTC m=+109.202406636" Jul 28 10:09:24.778506 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:24.778453 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-2wp29" podStartSLOduration=2.106233465 podStartE2EDuration="18.77843551s" totalImagesPullingTime="16.672202045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:24.777532746 +0000 UTC m=+109.220111716" watchObservedRunningTime="2026-07-28 10:09:24.77843551 +0000 UTC m=+109.221014486" Jul 28 10:09:29.637244 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.637201 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:29.637848 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.637316 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:29.642795 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.642770 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:29.762081 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.762043 2569 generic.go:356] "Generic (PLEG): container finished" podID="25239465-8177-4ac4-9cb3-a293eb83589e" containerID="b3874147583246bd15f4fc5e7249b8d831dfeab84bf3a6561891b1d92b6a5892" exitCode=0 Jul 28 10:09:29.762259 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.762125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" event={"ID":"25239465-8177-4ac4-9cb3-a293eb83589e","Type":"ContainerDied","Data":"b3874147583246bd15f4fc5e7249b8d831dfeab84bf3a6561891b1d92b6a5892"} Jul 28 10:09:29.762525 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.762506 2569 scope.go:117] "RemoveContainer" containerID="b3874147583246bd15f4fc5e7249b8d831dfeab84bf3a6561891b1d92b6a5892" Jul 28 10:09:29.763846 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.763819 2569 generic.go:356] "Generic (PLEG): container finished" podID="977426ec-297c-41ec-ab46-c83d7cf4d480" containerID="42cdadd63c0fd664aed8bd5e8f9b64a58aa383d5512a94319cddf30ae4b01ae7" exitCode=0 Jul 28 10:09:29.763980 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.763959 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" event={"ID":"977426ec-297c-41ec-ab46-c83d7cf4d480","Type":"ContainerDied","Data":"42cdadd63c0fd664aed8bd5e8f9b64a58aa383d5512a94319cddf30ae4b01ae7"} Jul 28 10:09:29.764387 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.764360 2569 scope.go:117] "RemoveContainer" containerID="42cdadd63c0fd664aed8bd5e8f9b64a58aa383d5512a94319cddf30ae4b01ae7" Jul 28 10:09:29.768696 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.768678 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:09:29.860092 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:29.860060 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64fb584c59-7lh2q"] Jul 28 10:09:30.771015 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:30.770975 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr" event={"ID":"25239465-8177-4ac4-9cb3-a293eb83589e","Type":"ContainerStarted","Data":"abe8767a965b8fe0043270aafd695d74632e2c9184b70ab73aad0eb8d7459e5f"} Jul 28 10:09:30.772977 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:30.772953 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rsw5m" event={"ID":"977426ec-297c-41ec-ab46-c83d7cf4d480","Type":"ContainerStarted","Data":"b1829e9af59ca3c4d2d3341c9249cffaea6e4c0923e60e8b83728c412cc49487"} Jul 28 10:09:36.828965 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:36.828921 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" podUID="03330411-ab4f-43a6-92f3-cefc7526cc9f" containerName="registry" containerID="cri-o://9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e" gracePeriod=30 Jul 28 10:09:37.113107 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.113040 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:09:37.275159 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275122 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-bound-sa-token\") pod \"03330411-ab4f-43a6-92f3-cefc7526cc9f\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " Jul 28 10:09:37.275372 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275191 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-certificates\") pod \"03330411-ab4f-43a6-92f3-cefc7526cc9f\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " Jul 28 10:09:37.275372 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275227 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") pod \"03330411-ab4f-43a6-92f3-cefc7526cc9f\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " Jul 28 10:09:37.275372 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275272 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-trusted-ca\") pod \"03330411-ab4f-43a6-92f3-cefc7526cc9f\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " Jul 28 10:09:37.275532 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275411 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03330411-ab4f-43a6-92f3-cefc7526cc9f-ca-trust-extracted\") pod \"03330411-ab4f-43a6-92f3-cefc7526cc9f\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " Jul 28 10:09:37.275532 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275501 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-image-registry-private-configuration\") pod \"03330411-ab4f-43a6-92f3-cefc7526cc9f\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " Jul 28 10:09:37.275655 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275538 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5psnq\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-kube-api-access-5psnq\") pod \"03330411-ab4f-43a6-92f3-cefc7526cc9f\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " Jul 28 10:09:37.275655 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275571 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-installation-pull-secrets\") pod \"03330411-ab4f-43a6-92f3-cefc7526cc9f\" (UID: \"03330411-ab4f-43a6-92f3-cefc7526cc9f\") " Jul 28 10:09:37.275757 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275704 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "03330411-ab4f-43a6-92f3-cefc7526cc9f" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:09:37.275812 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275758 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "03330411-ab4f-43a6-92f3-cefc7526cc9f" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:09:37.275972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275925 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-certificates\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:37.275972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.275948 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03330411-ab4f-43a6-92f3-cefc7526cc9f-trusted-ca\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:37.277726 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.277611 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "03330411-ab4f-43a6-92f3-cefc7526cc9f" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:09:37.277860 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.277841 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "03330411-ab4f-43a6-92f3-cefc7526cc9f" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:09:37.277972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.277948 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "03330411-ab4f-43a6-92f3-cefc7526cc9f" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:09:37.278109 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.278089 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "03330411-ab4f-43a6-92f3-cefc7526cc9f" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:09:37.278221 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.278199 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-kube-api-access-5psnq" (OuterVolumeSpecName: "kube-api-access-5psnq") pod "03330411-ab4f-43a6-92f3-cefc7526cc9f" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f"). InnerVolumeSpecName "kube-api-access-5psnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:09:37.283837 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.283814 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03330411-ab4f-43a6-92f3-cefc7526cc9f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "03330411-ab4f-43a6-92f3-cefc7526cc9f" (UID: "03330411-ab4f-43a6-92f3-cefc7526cc9f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:09:37.376880 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.376806 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-bound-sa-token\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:37.376880 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.376834 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-registry-tls\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:37.376880 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.376846 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03330411-ab4f-43a6-92f3-cefc7526cc9f-ca-trust-extracted\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:37.376880 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.376858 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-image-registry-private-configuration\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:37.376880 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.376872 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5psnq\" (UniqueName: \"kubernetes.io/projected/03330411-ab4f-43a6-92f3-cefc7526cc9f-kube-api-access-5psnq\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:37.377143 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.376885 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03330411-ab4f-43a6-92f3-cefc7526cc9f-installation-pull-secrets\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:37.802905 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.802868 2569 generic.go:356] "Generic (PLEG): container finished" podID="03330411-ab4f-43a6-92f3-cefc7526cc9f" containerID="9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e" exitCode=0 Jul 28 10:09:37.803187 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.802917 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" Jul 28 10:09:37.803187 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.802953 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" event={"ID":"03330411-ab4f-43a6-92f3-cefc7526cc9f","Type":"ContainerDied","Data":"9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e"} Jul 28 10:09:37.803187 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.803001 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659cc6d558-mxqwp" event={"ID":"03330411-ab4f-43a6-92f3-cefc7526cc9f","Type":"ContainerDied","Data":"1444023de931d654565782f5489d58701dfeee29724cea6d10cf26dc10d0253a"} Jul 28 10:09:37.803187 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.803020 2569 scope.go:117] "RemoveContainer" containerID="9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e" Jul 28 10:09:37.811472 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.811453 2569 scope.go:117] "RemoveContainer" containerID="9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e" Jul 28 10:09:37.811745 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:09:37.811726 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e\": container with ID starting with 9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e not found: ID does not exist" containerID="9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e" Jul 28 10:09:37.811800 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.811753 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e"} err="failed to get container status \"9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e\": rpc error: code = NotFound desc = could not find container \"9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e\": container with ID starting with 9f9acea3ed3d41977ae45ee81e73ebb4e1caf544731fdc53d8bb00591066f43e not found: ID does not exist" Jul 28 10:09:37.825435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.825411 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-659cc6d558-mxqwp"] Jul 28 10:09:37.829759 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:37.829739 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-659cc6d558-mxqwp"] Jul 28 10:09:38.078487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:38.078415 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03330411-ab4f-43a6-92f3-cefc7526cc9f" path="/var/lib/kubelet/pods/03330411-ab4f-43a6-92f3-cefc7526cc9f/volumes" Jul 28 10:09:39.812675 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:39.812637 2569 generic.go:356] "Generic (PLEG): container finished" podID="6ba36116-0fd9-4d91-b070-c52a5378dc54" containerID="56d9b7f4e82730b01c2f8e1658ee35a8907a1549d0de1175eff7c1daf0723ed8" exitCode=0 Jul 28 10:09:39.812675 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:39.812646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-jd9hz" event={"ID":"6ba36116-0fd9-4d91-b070-c52a5378dc54","Type":"ContainerDied","Data":"56d9b7f4e82730b01c2f8e1658ee35a8907a1549d0de1175eff7c1daf0723ed8"} Jul 28 10:09:39.813142 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:39.812968 2569 scope.go:117] "RemoveContainer" containerID="56d9b7f4e82730b01c2f8e1658ee35a8907a1549d0de1175eff7c1daf0723ed8" Jul 28 10:09:40.818323 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:40.818287 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-jd9hz" event={"ID":"6ba36116-0fd9-4d91-b070-c52a5378dc54","Type":"ContainerStarted","Data":"f51de292444ce471922aaca16f72258aa6dc6af4c7892e4b39d082a3f025e59f"} Jul 28 10:09:54.886171 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:54.886128 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64fb584c59-7lh2q" podUID="07155ef5-c784-42b7-be32-d8564c2afacd" containerName="console" containerID="cri-o://12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467" gracePeriod=15 Jul 28 10:09:55.164095 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.164071 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:55.324199 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.324159 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjqsp\" (UniqueName: \"kubernetes.io/projected/07155ef5-c784-42b7-be32-d8564c2afacd-kube-api-access-bjqsp\") pod \"07155ef5-c784-42b7-be32-d8564c2afacd\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " Jul 28 10:09:55.324199 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.324202 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-oauth-serving-cert\") pod \"07155ef5-c784-42b7-be32-d8564c2afacd\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " Jul 28 10:09:55.324412 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.324239 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-serving-cert\") pod \"07155ef5-c784-42b7-be32-d8564c2afacd\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " Jul 28 10:09:55.324412 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.324266 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-oauth-config\") pod \"07155ef5-c784-42b7-be32-d8564c2afacd\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " Jul 28 10:09:55.324412 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.324312 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-service-ca\") pod \"07155ef5-c784-42b7-be32-d8564c2afacd\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " Jul 28 10:09:55.324412 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.324368 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-console-config\") pod \"07155ef5-c784-42b7-be32-d8564c2afacd\" (UID: \"07155ef5-c784-42b7-be32-d8564c2afacd\") " Jul 28 10:09:55.324759 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.324602 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "07155ef5-c784-42b7-be32-d8564c2afacd" (UID: "07155ef5-c784-42b7-be32-d8564c2afacd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:09:55.324891 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.324765 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-service-ca" (OuterVolumeSpecName: "service-ca") pod "07155ef5-c784-42b7-be32-d8564c2afacd" (UID: "07155ef5-c784-42b7-be32-d8564c2afacd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:09:55.324891 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.324852 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-console-config" (OuterVolumeSpecName: "console-config") pod "07155ef5-c784-42b7-be32-d8564c2afacd" (UID: "07155ef5-c784-42b7-be32-d8564c2afacd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:09:55.326598 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.326569 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07155ef5-c784-42b7-be32-d8564c2afacd-kube-api-access-bjqsp" (OuterVolumeSpecName: "kube-api-access-bjqsp") pod "07155ef5-c784-42b7-be32-d8564c2afacd" (UID: "07155ef5-c784-42b7-be32-d8564c2afacd"). InnerVolumeSpecName "kube-api-access-bjqsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:09:55.326598 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.326585 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "07155ef5-c784-42b7-be32-d8564c2afacd" (UID: "07155ef5-c784-42b7-be32-d8564c2afacd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:09:55.326598 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.326601 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "07155ef5-c784-42b7-be32-d8564c2afacd" (UID: "07155ef5-c784-42b7-be32-d8564c2afacd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:09:55.425059 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.424966 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjqsp\" (UniqueName: \"kubernetes.io/projected/07155ef5-c784-42b7-be32-d8564c2afacd-kube-api-access-bjqsp\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:55.425059 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.424998 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-oauth-serving-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:55.425059 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.425011 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-serving-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:55.425059 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.425023 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07155ef5-c784-42b7-be32-d8564c2afacd-console-oauth-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:55.425059 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.425035 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-service-ca\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:55.425059 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.425047 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07155ef5-c784-42b7-be32-d8564c2afacd-console-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:09:55.883439 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.883406 2569 generic.go:356] "Generic (PLEG): container finished" podID="07155ef5-c784-42b7-be32-d8564c2afacd" containerID="12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467" exitCode=0 Jul 28 10:09:55.883588 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.883461 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64fb584c59-7lh2q" Jul 28 10:09:55.883588 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.883495 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64fb584c59-7lh2q" event={"ID":"07155ef5-c784-42b7-be32-d8564c2afacd","Type":"ContainerDied","Data":"12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467"} Jul 28 10:09:55.883588 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.883568 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64fb584c59-7lh2q" event={"ID":"07155ef5-c784-42b7-be32-d8564c2afacd","Type":"ContainerDied","Data":"b3d344c083be32463bead1071c228b0b3bba9aaabec884b8818f9083b90b78e8"} Jul 28 10:09:55.883729 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.883591 2569 scope.go:117] "RemoveContainer" containerID="12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467" Jul 28 10:09:55.892711 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.892475 2569 scope.go:117] "RemoveContainer" containerID="12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467" Jul 28 10:09:55.893029 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:09:55.892754 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467\": container with ID starting with 12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467 not found: ID does not exist" containerID="12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467" Jul 28 10:09:55.893029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.892779 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467"} err="failed to get container status \"12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467\": rpc error: code = NotFound desc = could not find container \"12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467\": container with ID starting with 12ae4cc28c13185f71f044f16d6a038ef46acab300899313d9c918a243905467 not found: ID does not exist" Jul 28 10:09:55.908504 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.908478 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64fb584c59-7lh2q"] Jul 28 10:09:55.912332 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:55.912301 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64fb584c59-7lh2q"] Jul 28 10:09:56.078183 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:09:56.078153 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07155ef5-c784-42b7-be32-d8564c2afacd" path="/var/lib/kubelet/pods/07155ef5-c784-42b7-be32-d8564c2afacd/volumes" Jul 28 10:10:27.013017 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.012979 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65ff56565c-jwvps"] Jul 28 10:10:27.013593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.013414 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03330411-ab4f-43a6-92f3-cefc7526cc9f" containerName="registry" Jul 28 10:10:27.013593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.013432 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="03330411-ab4f-43a6-92f3-cefc7526cc9f" containerName="registry" Jul 28 10:10:27.013593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.013446 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07155ef5-c784-42b7-be32-d8564c2afacd" containerName="console" Jul 28 10:10:27.013593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.013455 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="07155ef5-c784-42b7-be32-d8564c2afacd" containerName="console" Jul 28 10:10:27.013593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.013527 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="07155ef5-c784-42b7-be32-d8564c2afacd" containerName="console" Jul 28 10:10:27.013593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.013539 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="03330411-ab4f-43a6-92f3-cefc7526cc9f" containerName="registry" Jul 28 10:10:27.015903 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.015876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.029145 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.029119 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65ff56565c-jwvps"] Jul 28 10:10:27.062002 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.061971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-serving-cert\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.062002 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.062008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-trusted-ca-bundle\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.062255 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.062032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-oauth-config\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.062255 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.062162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nskz\" (UniqueName: \"kubernetes.io/projected/5b120778-154c-4396-8bfc-45966e9da270-kube-api-access-9nskz\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.062255 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.062219 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-console-config\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.062372 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.062251 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-oauth-serving-cert\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.062372 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.062320 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-service-ca\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.162749 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.162716 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-service-ca\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.162749 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.162760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-serving-cert\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.163005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.162785 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-trusted-ca-bundle\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.163005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.162811 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-oauth-config\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.163005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.162845 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nskz\" (UniqueName: \"kubernetes.io/projected/5b120778-154c-4396-8bfc-45966e9da270-kube-api-access-9nskz\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.163005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.162886 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-console-config\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.163005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.162909 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-oauth-serving-cert\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.163582 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.163548 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-service-ca\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.163856 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.163832 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-console-config\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.163939 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.163849 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-trusted-ca-bundle\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.163939 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.163915 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-oauth-serving-cert\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.165527 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.165495 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-oauth-config\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.165812 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.165795 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-serving-cert\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.171220 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.171200 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nskz\" (UniqueName: \"kubernetes.io/projected/5b120778-154c-4396-8bfc-45966e9da270-kube-api-access-9nskz\") pod \"console-65ff56565c-jwvps\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.328826 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.328740 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:27.461852 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:27.461826 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65ff56565c-jwvps"] Jul 28 10:10:27.463728 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:10:27.463692 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b120778_154c_4396_8bfc_45966e9da270.slice/crio-9b49cd3a53feb4b28c11fb8270831f62d765ce1690fbff9217852bb719cb0df3 WatchSource:0}: Error finding container 9b49cd3a53feb4b28c11fb8270831f62d765ce1690fbff9217852bb719cb0df3: Status 404 returned error can't find the container with id 9b49cd3a53feb4b28c11fb8270831f62d765ce1690fbff9217852bb719cb0df3 Jul 28 10:10:28.014584 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:28.014547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ff56565c-jwvps" event={"ID":"5b120778-154c-4396-8bfc-45966e9da270","Type":"ContainerStarted","Data":"a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3"} Jul 28 10:10:28.015046 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:28.014643 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ff56565c-jwvps" event={"ID":"5b120778-154c-4396-8bfc-45966e9da270","Type":"ContainerStarted","Data":"9b49cd3a53feb4b28c11fb8270831f62d765ce1690fbff9217852bb719cb0df3"} Jul 28 10:10:28.036153 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:28.036102 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65ff56565c-jwvps" podStartSLOduration=2.036088055 podStartE2EDuration="2.036088055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:10:28.034562401 +0000 UTC m=+172.477141377" watchObservedRunningTime="2026-07-28 10:10:28.036088055 +0000 UTC m=+172.478667028" Jul 28 10:10:37.329538 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:37.329504 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:37.329942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:37.329590 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:37.334130 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:37.334106 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:38.057456 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:38.057427 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:10:38.106790 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:10:38.106756 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8fb97b8f7-ppntn"] Jul 28 10:11:03.128671 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.128611 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8fb97b8f7-ppntn" podUID="44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" containerName="console" containerID="cri-o://fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582" gracePeriod=15 Jul 28 10:11:03.363999 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.363975 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:11:03.457880 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.457792 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-oauth-config\") pod \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " Jul 28 10:11:03.457880 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.457828 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-config\") pod \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " Jul 28 10:11:03.457880 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.457845 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-serving-cert\") pod \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " Jul 28 10:11:03.458161 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.457887 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75rp8\" (UniqueName: \"kubernetes.io/projected/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-kube-api-access-75rp8\") pod \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " Jul 28 10:11:03.458161 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.458002 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-service-ca\") pod \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " Jul 28 10:11:03.458161 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.458057 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-oauth-serving-cert\") pod \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " Jul 28 10:11:03.458161 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.458133 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-trusted-ca-bundle\") pod \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\" (UID: \"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412\") " Jul 28 10:11:03.458376 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.458339 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-config" (OuterVolumeSpecName: "console-config") pod "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" (UID: "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:11:03.458376 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.458363 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-service-ca" (OuterVolumeSpecName: "service-ca") pod "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" (UID: "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:11:03.458496 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.458421 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" (UID: "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:11:03.458700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.458680 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" (UID: "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:11:03.460006 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.459977 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" (UID: "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:11:03.460102 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.460027 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" (UID: "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:11:03.460102 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.460042 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-kube-api-access-75rp8" (OuterVolumeSpecName: "kube-api-access-75rp8") pod "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" (UID: "44f9bf1b-bff4-4253-a16b-1d7a7ccf5412"). InnerVolumeSpecName "kube-api-access-75rp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:11:03.559174 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.559135 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rp8\" (UniqueName: \"kubernetes.io/projected/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-kube-api-access-75rp8\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:11:03.559174 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.559167 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-service-ca\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:11:03.559174 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.559177 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-oauth-serving-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:11:03.559418 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.559186 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-trusted-ca-bundle\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:11:03.559418 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.559198 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-oauth-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:11:03.559418 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.559210 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:11:03.559418 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:03.559220 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412-console-serving-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:11:04.150694 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:04.150661 2569 generic.go:356] "Generic (PLEG): container finished" podID="44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" containerID="fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582" exitCode=0 Jul 28 10:11:04.151097 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:04.150705 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fb97b8f7-ppntn" Jul 28 10:11:04.151097 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:04.150747 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fb97b8f7-ppntn" event={"ID":"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412","Type":"ContainerDied","Data":"fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582"} Jul 28 10:11:04.151097 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:04.150797 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fb97b8f7-ppntn" event={"ID":"44f9bf1b-bff4-4253-a16b-1d7a7ccf5412","Type":"ContainerDied","Data":"a752c4ba0ec66421b15200635e466967b90ed3132aae695d41b30778f0b7a39f"} Jul 28 10:11:04.151097 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:04.150817 2569 scope.go:117] "RemoveContainer" containerID="fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582" Jul 28 10:11:04.158847 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:04.158835 2569 scope.go:117] "RemoveContainer" containerID="fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582" Jul 28 10:11:04.159089 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:11:04.159069 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582\": container with ID starting with fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582 not found: ID does not exist" containerID="fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582" Jul 28 10:11:04.159140 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:04.159098 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582"} err="failed to get container status \"fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582\": rpc error: code = NotFound desc = could not find container \"fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582\": container with ID starting with fc796a7a9ab670e815abf7808dc4ae1f86fca293c674ad337443981097445582 not found: ID does not exist" Jul 28 10:11:04.168279 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:04.168254 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8fb97b8f7-ppntn"] Jul 28 10:11:04.173274 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:04.173254 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8fb97b8f7-ppntn"] Jul 28 10:11:06.078230 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:11:06.078191 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" path="/var/lib/kubelet/pods/44f9bf1b-bff4-4253-a16b-1d7a7ccf5412/volumes" Jul 28 10:12:06.718634 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.718537 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b78c8cdcf-rwj92"] Jul 28 10:12:06.719142 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.718841 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" containerName="console" Jul 28 10:12:06.719142 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.718854 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" containerName="console" Jul 28 10:12:06.719142 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.718923 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f9bf1b-bff4-4253-a16b-1d7a7ccf5412" containerName="console" Jul 28 10:12:06.721002 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.720980 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.730356 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.730331 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b78c8cdcf-rwj92"] Jul 28 10:12:06.751176 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.751140 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-trusted-ca-bundle\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.751403 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.751192 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-service-ca\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.751403 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.751256 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrhb\" (UniqueName: \"kubernetes.io/projected/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-kube-api-access-6hrhb\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.751403 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.751313 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-oauth-serving-cert\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.751403 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.751338 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-config\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.751974 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.751423 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-serving-cert\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.752095 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.752008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-oauth-config\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.852556 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.852515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-config\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.852746 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.852573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-serving-cert\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.852746 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.852603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-oauth-config\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.852746 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.852644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-trusted-ca-bundle\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.852746 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.852670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-service-ca\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.852746 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.852700 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrhb\" (UniqueName: \"kubernetes.io/projected/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-kube-api-access-6hrhb\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.852994 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.852756 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-oauth-serving-cert\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.853474 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.853447 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-config\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.853474 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.853462 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-service-ca\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.853616 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.853474 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-oauth-serving-cert\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.853825 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.853802 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-trusted-ca-bundle\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.855151 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.855125 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-oauth-config\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.855354 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.855331 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-serving-cert\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:06.861004 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:06.860984 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrhb\" (UniqueName: \"kubernetes.io/projected/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-kube-api-access-6hrhb\") pod \"console-b78c8cdcf-rwj92\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:07.031079 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:07.031035 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:07.161808 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:07.161782 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b78c8cdcf-rwj92"] Jul 28 10:12:07.164400 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:12:07.164377 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd608241e_e3ab_4d43_bd7c_b80481b5d9f7.slice/crio-938563f06c3dc698397cffe3ccb822cb4d33e78cd13bf1ffe2e311de9655159c WatchSource:0}: Error finding container 938563f06c3dc698397cffe3ccb822cb4d33e78cd13bf1ffe2e311de9655159c: Status 404 returned error can't find the container with id 938563f06c3dc698397cffe3ccb822cb4d33e78cd13bf1ffe2e311de9655159c Jul 28 10:12:07.388180 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:07.388094 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78c8cdcf-rwj92" event={"ID":"d608241e-e3ab-4d43-bd7c-b80481b5d9f7","Type":"ContainerStarted","Data":"10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4"} Jul 28 10:12:07.388180 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:07.388143 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78c8cdcf-rwj92" event={"ID":"d608241e-e3ab-4d43-bd7c-b80481b5d9f7","Type":"ContainerStarted","Data":"938563f06c3dc698397cffe3ccb822cb4d33e78cd13bf1ffe2e311de9655159c"} Jul 28 10:12:07.408418 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:07.408369 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b78c8cdcf-rwj92" podStartSLOduration=1.408354094 podStartE2EDuration="1.408354094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:12:07.40713796 +0000 UTC m=+271.849716935" watchObservedRunningTime="2026-07-28 10:12:07.408354094 +0000 UTC m=+271.850933068" Jul 28 10:12:17.032022 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:17.031976 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:17.032502 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:17.032048 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:17.036721 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:17.036688 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:17.430005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:17.429924 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:12:17.484424 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:17.484389 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65ff56565c-jwvps"] Jul 28 10:12:35.915882 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:35.915852 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:12:35.921391 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:35.921368 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:12:35.929863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:35.929843 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 10:12:42.509843 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.509797 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65ff56565c-jwvps" podUID="5b120778-154c-4396-8bfc-45966e9da270" containerName="console" containerID="cri-o://a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3" gracePeriod=15 Jul 28 10:12:42.747767 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.747745 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:12:42.846489 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.846410 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nskz\" (UniqueName: \"kubernetes.io/projected/5b120778-154c-4396-8bfc-45966e9da270-kube-api-access-9nskz\") pod \"5b120778-154c-4396-8bfc-45966e9da270\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " Jul 28 10:12:42.846489 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.846483 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-oauth-serving-cert\") pod \"5b120778-154c-4396-8bfc-45966e9da270\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " Jul 28 10:12:42.846714 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.846648 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-console-config\") pod \"5b120778-154c-4396-8bfc-45966e9da270\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " Jul 28 10:12:42.846714 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.846708 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-oauth-config\") pod \"5b120778-154c-4396-8bfc-45966e9da270\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " Jul 28 10:12:42.846830 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.846732 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-serving-cert\") pod \"5b120778-154c-4396-8bfc-45966e9da270\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " Jul 28 10:12:42.846830 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.846761 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-service-ca\") pod \"5b120778-154c-4396-8bfc-45966e9da270\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " Jul 28 10:12:42.846830 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.846814 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b120778-154c-4396-8bfc-45966e9da270" (UID: "5b120778-154c-4396-8bfc-45966e9da270"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:12:42.846979 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.846872 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-trusted-ca-bundle\") pod \"5b120778-154c-4396-8bfc-45966e9da270\" (UID: \"5b120778-154c-4396-8bfc-45966e9da270\") " Jul 28 10:12:42.847074 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.846947 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-console-config" (OuterVolumeSpecName: "console-config") pod "5b120778-154c-4396-8bfc-45966e9da270" (UID: "5b120778-154c-4396-8bfc-45966e9da270"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:12:42.847144 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.847125 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b120778-154c-4396-8bfc-45966e9da270" (UID: "5b120778-154c-4396-8bfc-45966e9da270"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:12:42.847254 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.847234 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-oauth-serving-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:12:42.847306 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.847261 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-console-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:12:42.847306 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.847277 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-service-ca\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:12:42.847424 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.847396 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5b120778-154c-4396-8bfc-45966e9da270" (UID: "5b120778-154c-4396-8bfc-45966e9da270"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:12:42.848631 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.848597 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b120778-154c-4396-8bfc-45966e9da270-kube-api-access-9nskz" (OuterVolumeSpecName: "kube-api-access-9nskz") pod "5b120778-154c-4396-8bfc-45966e9da270" (UID: "5b120778-154c-4396-8bfc-45966e9da270"). InnerVolumeSpecName "kube-api-access-9nskz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:12:42.848778 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.848759 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b120778-154c-4396-8bfc-45966e9da270" (UID: "5b120778-154c-4396-8bfc-45966e9da270"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:12:42.848839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.848772 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b120778-154c-4396-8bfc-45966e9da270" (UID: "5b120778-154c-4396-8bfc-45966e9da270"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:12:42.948596 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.948560 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-oauth-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:12:42.948596 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.948588 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b120778-154c-4396-8bfc-45966e9da270-console-serving-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:12:42.948596 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.948599 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b120778-154c-4396-8bfc-45966e9da270-trusted-ca-bundle\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:12:42.948836 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:42.948608 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nskz\" (UniqueName: \"kubernetes.io/projected/5b120778-154c-4396-8bfc-45966e9da270-kube-api-access-9nskz\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:12:43.527697 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:43.527658 2569 generic.go:356] "Generic (PLEG): container finished" podID="5b120778-154c-4396-8bfc-45966e9da270" containerID="a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3" exitCode=0 Jul 28 10:12:43.528170 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:43.527715 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ff56565c-jwvps" Jul 28 10:12:43.528170 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:43.527748 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ff56565c-jwvps" event={"ID":"5b120778-154c-4396-8bfc-45966e9da270","Type":"ContainerDied","Data":"a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3"} Jul 28 10:12:43.528170 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:43.527801 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ff56565c-jwvps" event={"ID":"5b120778-154c-4396-8bfc-45966e9da270","Type":"ContainerDied","Data":"9b49cd3a53feb4b28c11fb8270831f62d765ce1690fbff9217852bb719cb0df3"} Jul 28 10:12:43.528170 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:43.527823 2569 scope.go:117] "RemoveContainer" containerID="a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3" Jul 28 10:12:43.536651 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:43.536603 2569 scope.go:117] "RemoveContainer" containerID="a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3" Jul 28 10:12:43.536907 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:12:43.536887 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3\": container with ID starting with a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3 not found: ID does not exist" containerID="a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3" Jul 28 10:12:43.536972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:43.536919 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3"} err="failed to get container status \"a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3\": rpc error: code = NotFound desc = could not find container \"a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3\": container with ID starting with a5d681363b40b1940080a5472b550533695521a82444a6140f9390630b2d27e3 not found: ID does not exist" Jul 28 10:12:43.549839 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:43.549809 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65ff56565c-jwvps"] Jul 28 10:12:43.553560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:43.553539 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65ff56565c-jwvps"] Jul 28 10:12:44.078397 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:12:44.078369 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b120778-154c-4396-8bfc-45966e9da270" path="/var/lib/kubelet/pods/5b120778-154c-4396-8bfc-45966e9da270/volumes" Jul 28 10:14:06.596575 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.596543 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-kd6r4"] Jul 28 10:14:06.597002 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.596874 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b120778-154c-4396-8bfc-45966e9da270" containerName="console" Jul 28 10:14:06.597002 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.596888 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b120778-154c-4396-8bfc-45966e9da270" containerName="console" Jul 28 10:14:06.597002 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.596952 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b120778-154c-4396-8bfc-45966e9da270" containerName="console" Jul 28 10:14:06.598642 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.598611 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:06.601923 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.601900 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 28 10:14:06.602052 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.601912 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 28 10:14:06.602052 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.601912 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 28 10:14:06.602052 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.601980 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-s8l5h\"" Jul 28 10:14:06.602052 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.601933 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Jul 28 10:14:06.603169 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.603149 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 28 10:14:06.608078 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.608053 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-kd6r4"] Jul 28 10:14:06.681259 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.681227 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/425fc0c3-8971-4679-b11b-2c38edd1054a-cabundle0\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:06.681409 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.681274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/425fc0c3-8971-4679-b11b-2c38edd1054a-certificates\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:06.681409 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.681300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6dbx\" (UniqueName: \"kubernetes.io/projected/425fc0c3-8971-4679-b11b-2c38edd1054a-kube-api-access-v6dbx\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:06.782548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.782512 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/425fc0c3-8971-4679-b11b-2c38edd1054a-certificates\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:06.782548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.782553 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6dbx\" (UniqueName: \"kubernetes.io/projected/425fc0c3-8971-4679-b11b-2c38edd1054a-kube-api-access-v6dbx\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:06.782767 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.782616 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/425fc0c3-8971-4679-b11b-2c38edd1054a-cabundle0\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:06.782767 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:14:06.782690 2569 secret.go:281] references non-existent secret key: ca.crt Jul 28 10:14:06.782767 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:14:06.782708 2569 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Jul 28 10:14:06.782767 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:14:06.782719 2569 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-kd6r4: references non-existent secret key: ca.crt Jul 28 10:14:06.782889 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:14:06.782793 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/425fc0c3-8971-4679-b11b-2c38edd1054a-certificates podName:425fc0c3-8971-4679-b11b-2c38edd1054a nodeName:}" failed. No retries permitted until 2026-07-28 10:14:07.282774136 +0000 UTC m=+391.725353109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/425fc0c3-8971-4679-b11b-2c38edd1054a-certificates") pod "keda-operator-77f648d694-kd6r4" (UID: "425fc0c3-8971-4679-b11b-2c38edd1054a") : references non-existent secret key: ca.crt Jul 28 10:14:06.783301 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.783283 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/425fc0c3-8971-4679-b11b-2c38edd1054a-cabundle0\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:06.793615 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.793594 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6dbx\" (UniqueName: \"kubernetes.io/projected/425fc0c3-8971-4679-b11b-2c38edd1054a-kube-api-access-v6dbx\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:06.796773 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.796753 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv"] Jul 28 10:14:06.798996 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.798976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:06.801787 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.801768 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Jul 28 10:14:06.807602 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.807569 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv"] Jul 28 10:14:06.884097 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.884016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/dcac1035-066f-4c02-9b6a-682a734f808d-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:06.884097 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.884059 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dcac1035-066f-4c02-9b6a-682a734f808d-certificates\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:06.884294 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.884166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnv4f\" (UniqueName: \"kubernetes.io/projected/dcac1035-066f-4c02-9b6a-682a734f808d-kube-api-access-xnv4f\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:06.984833 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.984798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnv4f\" (UniqueName: \"kubernetes.io/projected/dcac1035-066f-4c02-9b6a-682a734f808d-kube-api-access-xnv4f\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:06.984976 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.984862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/dcac1035-066f-4c02-9b6a-682a734f808d-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:06.984976 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.984900 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dcac1035-066f-4c02-9b6a-682a734f808d-certificates\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:06.985054 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:14:06.985007 2569 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found Jul 28 10:14:06.985054 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:14:06.985027 2569 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv: secret "keda-metrics-apiserver-certs" not found Jul 28 10:14:06.985124 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:14:06.985091 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dcac1035-066f-4c02-9b6a-682a734f808d-certificates podName:dcac1035-066f-4c02-9b6a-682a734f808d nodeName:}" failed. No retries permitted until 2026-07-28 10:14:07.485071423 +0000 UTC m=+391.927650385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/dcac1035-066f-4c02-9b6a-682a734f808d-certificates") pod "keda-metrics-apiserver-55557df8c-9zbjv" (UID: "dcac1035-066f-4c02-9b6a-682a734f808d") : secret "keda-metrics-apiserver-certs" not found Jul 28 10:14:06.985252 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.985234 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/dcac1035-066f-4c02-9b6a-682a734f808d-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:06.999570 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:06.999543 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnv4f\" (UniqueName: \"kubernetes.io/projected/dcac1035-066f-4c02-9b6a-682a734f808d-kube-api-access-xnv4f\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:07.011864 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.011838 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-tc9v4"] Jul 28 10:14:07.013863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.013848 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:14:07.016607 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.016587 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 28 10:14:07.022653 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.022615 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-tc9v4"] Jul 28 10:14:07.186058 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.185954 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz8x4\" (UniqueName: \"kubernetes.io/projected/b87afce6-d744-40d8-80d2-f2810755c50c-kube-api-access-gz8x4\") pod \"keda-admission-56f9798bb4-tc9v4\" (UID: \"b87afce6-d744-40d8-80d2-f2810755c50c\") " pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:14:07.186058 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.185998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b87afce6-d744-40d8-80d2-f2810755c50c-certificates\") pod \"keda-admission-56f9798bb4-tc9v4\" (UID: \"b87afce6-d744-40d8-80d2-f2810755c50c\") " pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:14:07.286937 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.286901 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/425fc0c3-8971-4679-b11b-2c38edd1054a-certificates\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:07.287111 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.286951 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8x4\" (UniqueName: \"kubernetes.io/projected/b87afce6-d744-40d8-80d2-f2810755c50c-kube-api-access-gz8x4\") pod \"keda-admission-56f9798bb4-tc9v4\" (UID: \"b87afce6-d744-40d8-80d2-f2810755c50c\") " pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:14:07.287111 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.286993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b87afce6-d744-40d8-80d2-f2810755c50c-certificates\") pod \"keda-admission-56f9798bb4-tc9v4\" (UID: \"b87afce6-d744-40d8-80d2-f2810755c50c\") " pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:14:07.289726 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.289699 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/425fc0c3-8971-4679-b11b-2c38edd1054a-certificates\") pod \"keda-operator-77f648d694-kd6r4\" (UID: \"425fc0c3-8971-4679-b11b-2c38edd1054a\") " pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:07.290783 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.290759 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b87afce6-d744-40d8-80d2-f2810755c50c-certificates\") pod \"keda-admission-56f9798bb4-tc9v4\" (UID: \"b87afce6-d744-40d8-80d2-f2810755c50c\") " pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:14:07.296581 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.296557 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8x4\" (UniqueName: \"kubernetes.io/projected/b87afce6-d744-40d8-80d2-f2810755c50c-kube-api-access-gz8x4\") pod \"keda-admission-56f9798bb4-tc9v4\" (UID: \"b87afce6-d744-40d8-80d2-f2810755c50c\") " pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:14:07.325263 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.325024 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:14:07.465424 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.465336 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-tc9v4"] Jul 28 10:14:07.471124 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:14:07.471097 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87afce6_d744_40d8_80d2_f2810755c50c.slice/crio-75983f4f6670fe0cbd6ffcd61d88c31b9a2eed7d4363cef3d80b8f5912a53a7b WatchSource:0}: Error finding container 75983f4f6670fe0cbd6ffcd61d88c31b9a2eed7d4363cef3d80b8f5912a53a7b: Status 404 returned error can't find the container with id 75983f4f6670fe0cbd6ffcd61d88c31b9a2eed7d4363cef3d80b8f5912a53a7b Jul 28 10:14:07.472405 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.472387 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 10:14:07.489268 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.489238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dcac1035-066f-4c02-9b6a-682a734f808d-certificates\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:07.491548 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.491524 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/dcac1035-066f-4c02-9b6a-682a734f808d-certificates\") pod \"keda-metrics-apiserver-55557df8c-9zbjv\" (UID: \"dcac1035-066f-4c02-9b6a-682a734f808d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:07.510853 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.510831 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:07.641106 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.640936 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-kd6r4"] Jul 28 10:14:07.643535 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:14:07.643500 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425fc0c3_8971_4679_b11b_2c38edd1054a.slice/crio-78f45197cdd626bc26619b00fb1544c35a8b6949881b616d702892d17d7b040a WatchSource:0}: Error finding container 78f45197cdd626bc26619b00fb1544c35a8b6949881b616d702892d17d7b040a: Status 404 returned error can't find the container with id 78f45197cdd626bc26619b00fb1544c35a8b6949881b616d702892d17d7b040a Jul 28 10:14:07.709971 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.709935 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:07.848250 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.848215 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv"] Jul 28 10:14:07.851839 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:14:07.851812 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcac1035_066f_4c02_9b6a_682a734f808d.slice/crio-cfb1eefd94db32b7cb7f93683f967cabf1dcbff6539d18f7c294c57041e5462a WatchSource:0}: Error finding container cfb1eefd94db32b7cb7f93683f967cabf1dcbff6539d18f7c294c57041e5462a: Status 404 returned error can't find the container with id cfb1eefd94db32b7cb7f93683f967cabf1dcbff6539d18f7c294c57041e5462a Jul 28 10:14:07.854900 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.854861 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" event={"ID":"b87afce6-d744-40d8-80d2-f2810755c50c","Type":"ContainerStarted","Data":"75983f4f6670fe0cbd6ffcd61d88c31b9a2eed7d4363cef3d80b8f5912a53a7b"} Jul 28 10:14:07.855993 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:07.855970 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-kd6r4" event={"ID":"425fc0c3-8971-4679-b11b-2c38edd1054a","Type":"ContainerStarted","Data":"78f45197cdd626bc26619b00fb1544c35a8b6949881b616d702892d17d7b040a"} Jul 28 10:14:08.866441 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:08.866397 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" event={"ID":"dcac1035-066f-4c02-9b6a-682a734f808d","Type":"ContainerStarted","Data":"cfb1eefd94db32b7cb7f93683f967cabf1dcbff6539d18f7c294c57041e5462a"} Jul 28 10:14:12.885841 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:12.885798 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-kd6r4" event={"ID":"425fc0c3-8971-4679-b11b-2c38edd1054a","Type":"ContainerStarted","Data":"00bc39dbe956973ccb25994e03b03e089047047fa5b72eccd658b04b6d7c8b6e"} Jul 28 10:14:12.886315 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:12.885930 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:12.887351 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:12.887327 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" event={"ID":"dcac1035-066f-4c02-9b6a-682a734f808d","Type":"ContainerStarted","Data":"d1bd61b3c319337aea0d4235643215b487a99238d9dc1f5062231a8c3b30481c"} Jul 28 10:14:12.887461 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:12.887418 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:12.888771 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:12.888747 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" event={"ID":"b87afce6-d744-40d8-80d2-f2810755c50c","Type":"ContainerStarted","Data":"dc29cbac616596ab2e04130419e619122d605d93ca44301458dc13009283ebba"} Jul 28 10:14:12.888894 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:12.888880 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:14:12.906085 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:12.906043 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-kd6r4" podStartSLOduration=2.031794038 podStartE2EDuration="6.906031348s" totalImagesPullingTime="4.87423731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:14:12.904859382 +0000 UTC m=+397.347438362" watchObservedRunningTime="2026-07-28 10:14:12.906031348 +0000 UTC m=+397.348610323" Jul 28 10:14:12.921700 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:12.921652 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" podStartSLOduration=2.25924137 podStartE2EDuration="6.921641566s" totalImagesPullingTime="4.662400196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:14:12.92094159 +0000 UTC m=+397.363520591" watchObservedRunningTime="2026-07-28 10:14:12.921641566 +0000 UTC m=+397.364220532" Jul 28 10:14:12.937306 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:12.937259 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" podStartSLOduration=1.904816777 podStartE2EDuration="6.937242944s" totalImagesPullingTime="5.032426167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:14:12.936688185 +0000 UTC m=+397.379267161" watchObservedRunningTime="2026-07-28 10:14:12.937242944 +0000 UTC m=+397.379821920" Jul 28 10:14:23.896753 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:23.896720 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-9zbjv" Jul 28 10:14:33.895044 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:33.895011 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-kd6r4" Jul 28 10:14:33.895462 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:14:33.895252 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-tc9v4" Jul 28 10:15:26.297983 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.297950 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6968749498-l45v7"] Jul 28 10:15:26.300143 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.300128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:15:26.303099 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.303072 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-jwcg4\"" Jul 28 10:15:26.303099 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.303074 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 28 10:15:26.303273 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.303114 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 28 10:15:26.304525 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.304503 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 28 10:15:26.309401 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.309380 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-l45v7"] Jul 28 10:15:26.333187 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.333167 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-jqhcq"] Jul 28 10:15:26.334414 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.334388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwmd\" (UniqueName: \"kubernetes.io/projected/1e3ad91d-6aac-4486-944c-e7a9e1e57708-kube-api-access-4nwmd\") pod \"kserve-controller-manager-6968749498-l45v7\" (UID: \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\") " pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:15:26.334516 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.334435 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3ad91d-6aac-4486-944c-e7a9e1e57708-cert\") pod \"kserve-controller-manager-6968749498-l45v7\" (UID: \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\") " pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:15:26.335252 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.335238 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:26.337779 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.337761 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 28 10:15:26.337878 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.337828 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2wxxj\"" Jul 28 10:15:26.344059 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.344040 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jqhcq"] Jul 28 10:15:26.435305 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.435276 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbptq\" (UniqueName: \"kubernetes.io/projected/22c7897f-0234-4331-99f3-0e931bffcc4b-kube-api-access-xbptq\") pod \"seaweedfs-64568bcd49-jqhcq\" (UID: \"22c7897f-0234-4331-99f3-0e931bffcc4b\") " pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:26.435484 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.435336 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwmd\" (UniqueName: \"kubernetes.io/projected/1e3ad91d-6aac-4486-944c-e7a9e1e57708-kube-api-access-4nwmd\") pod \"kserve-controller-manager-6968749498-l45v7\" (UID: \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\") " pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:15:26.435484 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.435374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3ad91d-6aac-4486-944c-e7a9e1e57708-cert\") pod \"kserve-controller-manager-6968749498-l45v7\" (UID: \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\") " pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:15:26.435484 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.435402 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22c7897f-0234-4331-99f3-0e931bffcc4b-data\") pod \"seaweedfs-64568bcd49-jqhcq\" (UID: \"22c7897f-0234-4331-99f3-0e931bffcc4b\") " pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:26.437842 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.437818 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3ad91d-6aac-4486-944c-e7a9e1e57708-cert\") pod \"kserve-controller-manager-6968749498-l45v7\" (UID: \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\") " pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:15:26.443091 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.443069 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwmd\" (UniqueName: \"kubernetes.io/projected/1e3ad91d-6aac-4486-944c-e7a9e1e57708-kube-api-access-4nwmd\") pod \"kserve-controller-manager-6968749498-l45v7\" (UID: \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\") " pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:15:26.536159 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.536131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbptq\" (UniqueName: \"kubernetes.io/projected/22c7897f-0234-4331-99f3-0e931bffcc4b-kube-api-access-xbptq\") pod \"seaweedfs-64568bcd49-jqhcq\" (UID: \"22c7897f-0234-4331-99f3-0e931bffcc4b\") " pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:26.536291 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.536197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22c7897f-0234-4331-99f3-0e931bffcc4b-data\") pod \"seaweedfs-64568bcd49-jqhcq\" (UID: \"22c7897f-0234-4331-99f3-0e931bffcc4b\") " pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:26.536535 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.536518 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22c7897f-0234-4331-99f3-0e931bffcc4b-data\") pod \"seaweedfs-64568bcd49-jqhcq\" (UID: \"22c7897f-0234-4331-99f3-0e931bffcc4b\") " pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:26.545934 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.545913 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbptq\" (UniqueName: \"kubernetes.io/projected/22c7897f-0234-4331-99f3-0e931bffcc4b-kube-api-access-xbptq\") pod \"seaweedfs-64568bcd49-jqhcq\" (UID: \"22c7897f-0234-4331-99f3-0e931bffcc4b\") " pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:26.611167 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.611076 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:15:26.644437 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.644407 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:26.748127 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.748102 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-l45v7"] Jul 28 10:15:26.751654 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:15:26.751606 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3ad91d_6aac_4486_944c_e7a9e1e57708.slice/crio-e99ffd2cf0ed257f9fa34405c9b5a239a52bbcf5e2bb3097f66cb27e5c6a72d0 WatchSource:0}: Error finding container e99ffd2cf0ed257f9fa34405c9b5a239a52bbcf5e2bb3097f66cb27e5c6a72d0: Status 404 returned error can't find the container with id e99ffd2cf0ed257f9fa34405c9b5a239a52bbcf5e2bb3097f66cb27e5c6a72d0 Jul 28 10:15:26.781477 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:26.781456 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jqhcq"] Jul 28 10:15:26.783928 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:15:26.783903 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c7897f_0234_4331_99f3_0e931bffcc4b.slice/crio-f743f9a638e89b1caa61fedd626cfde584080aff7dc690609e7909df91dff0eb WatchSource:0}: Error finding container f743f9a638e89b1caa61fedd626cfde584080aff7dc690609e7909df91dff0eb: Status 404 returned error can't find the container with id f743f9a638e89b1caa61fedd626cfde584080aff7dc690609e7909df91dff0eb Jul 28 10:15:27.174130 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.174088 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-l45v7" event={"ID":"1e3ad91d-6aac-4486-944c-e7a9e1e57708","Type":"ContainerStarted","Data":"e99ffd2cf0ed257f9fa34405c9b5a239a52bbcf5e2bb3097f66cb27e5c6a72d0"} Jul 28 10:15:27.175563 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.175527 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jqhcq" event={"ID":"22c7897f-0234-4331-99f3-0e931bffcc4b","Type":"ContainerStarted","Data":"f743f9a638e89b1caa61fedd626cfde584080aff7dc690609e7909df91dff0eb"} Jul 28 10:15:27.400964 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.399346 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65fc554bd4-dcwpw"] Jul 28 10:15:27.402280 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.402254 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.415567 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.415412 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fc554bd4-dcwpw"] Jul 28 10:15:27.454029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.446738 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e659632-57f5-4774-a888-145008a48b20-console-oauth-config\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.454029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.446800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-trusted-ca-bundle\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.454029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.446823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-oauth-serving-cert\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.454029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.446879 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75p9b\" (UniqueName: \"kubernetes.io/projected/6e659632-57f5-4774-a888-145008a48b20-kube-api-access-75p9b\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.454029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.446908 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-console-config\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.454029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.446947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-service-ca\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.454029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.446984 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e659632-57f5-4774-a888-145008a48b20-console-serving-cert\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.548552 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.548174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-service-ca\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.548552 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.548231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e659632-57f5-4774-a888-145008a48b20-console-serving-cert\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.548552 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.548281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e659632-57f5-4774-a888-145008a48b20-console-oauth-config\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.548552 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.548329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-trusted-ca-bundle\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.548552 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.548352 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-oauth-serving-cert\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.548552 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.548414 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75p9b\" (UniqueName: \"kubernetes.io/projected/6e659632-57f5-4774-a888-145008a48b20-kube-api-access-75p9b\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.548552 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.548444 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-console-config\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.549054 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.548994 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-service-ca\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.549359 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.549304 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-oauth-serving-cert\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.549485 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.549368 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-console-config\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.551074 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.550249 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e659632-57f5-4774-a888-145008a48b20-trusted-ca-bundle\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.552205 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.552164 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e659632-57f5-4774-a888-145008a48b20-console-oauth-config\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.552955 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.552920 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e659632-57f5-4774-a888-145008a48b20-console-serving-cert\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.568242 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.568218 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75p9b\" (UniqueName: \"kubernetes.io/projected/6e659632-57f5-4774-a888-145008a48b20-kube-api-access-75p9b\") pod \"console-65fc554bd4-dcwpw\" (UID: \"6e659632-57f5-4774-a888-145008a48b20\") " pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.719577 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.719373 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:27.902108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:27.902001 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fc554bd4-dcwpw"] Jul 28 10:15:27.906241 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:15:27.905397 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e659632_57f5_4774_a888_145008a48b20.slice/crio-901060736eea762d8cec20f5deec81cb596b0c1bc9aedfb0afa32c7f3c430434 WatchSource:0}: Error finding container 901060736eea762d8cec20f5deec81cb596b0c1bc9aedfb0afa32c7f3c430434: Status 404 returned error can't find the container with id 901060736eea762d8cec20f5deec81cb596b0c1bc9aedfb0afa32c7f3c430434 Jul 28 10:15:28.186116 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:28.186076 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fc554bd4-dcwpw" event={"ID":"6e659632-57f5-4774-a888-145008a48b20","Type":"ContainerStarted","Data":"9955a5d7b5518ed4a5e40b4312437208c93ba578cebefee3621afda17fcbc797"} Jul 28 10:15:28.186298 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:28.186133 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fc554bd4-dcwpw" event={"ID":"6e659632-57f5-4774-a888-145008a48b20","Type":"ContainerStarted","Data":"901060736eea762d8cec20f5deec81cb596b0c1bc9aedfb0afa32c7f3c430434"} Jul 28 10:15:28.208487 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:28.208291 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65fc554bd4-dcwpw" podStartSLOduration=1.208269803 podStartE2EDuration="1.208269803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:15:28.204832873 +0000 UTC m=+472.647411848" watchObservedRunningTime="2026-07-28 10:15:28.208269803 +0000 UTC m=+472.650848779" Jul 28 10:15:31.202564 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:31.202526 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-l45v7" event={"ID":"1e3ad91d-6aac-4486-944c-e7a9e1e57708","Type":"ContainerStarted","Data":"a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667"} Jul 28 10:15:31.203017 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:31.202592 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:15:31.204108 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:31.204086 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jqhcq" event={"ID":"22c7897f-0234-4331-99f3-0e931bffcc4b","Type":"ContainerStarted","Data":"1289ab27a73b13b5f11d6c6348a1eab21408e52a9610db8e0a111fb33e1afbf5"} Jul 28 10:15:31.204186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:31.204145 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:31.218815 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:31.218773 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6968749498-l45v7" podStartSLOduration=1.6469647250000001 podStartE2EDuration="5.21875949s" totalImagesPullingTime="3.571794765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:15:31.218744517 +0000 UTC m=+475.661323493" watchObservedRunningTime="2026-07-28 10:15:31.21875949 +0000 UTC m=+475.661338464" Jul 28 10:15:31.234193 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:31.234149 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-jqhcq" podStartSLOduration=1.626736925 podStartE2EDuration="5.234137716s" totalImagesPullingTime="3.607400791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:15:31.233802167 +0000 UTC m=+475.676381141" watchObservedRunningTime="2026-07-28 10:15:31.234137716 +0000 UTC m=+475.676716691" Jul 28 10:15:37.211463 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:37.211429 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-jqhcq" Jul 28 10:15:37.719687 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:37.719645 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:37.719864 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:37.719726 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:37.724384 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:37.724361 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:38.235522 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:38.235494 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65fc554bd4-dcwpw" Jul 28 10:15:38.283942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:15:38.283912 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b78c8cdcf-rwj92"] Jul 28 10:16:02.214555 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:02.214526 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:16:03.306846 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.306776 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b78c8cdcf-rwj92" podUID="d608241e-e3ab-4d43-bd7c-b80481b5d9f7" containerName="console" containerID="cri-o://10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4" gracePeriod=15 Jul 28 10:16:03.570323 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.570248 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:16:03.659645 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.659591 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-config\") pod \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " Jul 28 10:16:03.659645 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.659650 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-service-ca\") pod \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " Jul 28 10:16:03.659851 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.659685 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-trusted-ca-bundle\") pod \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " Jul 28 10:16:03.659851 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.659783 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-oauth-config\") pod \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " Jul 28 10:16:03.659937 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.659863 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-oauth-serving-cert\") pod \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " Jul 28 10:16:03.659937 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.659890 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-serving-cert\") pod \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " Jul 28 10:16:03.660153 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.660124 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-config" (OuterVolumeSpecName: "console-config") pod "d608241e-e3ab-4d43-bd7c-b80481b5d9f7" (UID: "d608241e-e3ab-4d43-bd7c-b80481b5d9f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:16:03.660219 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.660148 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "d608241e-e3ab-4d43-bd7c-b80481b5d9f7" (UID: "d608241e-e3ab-4d43-bd7c-b80481b5d9f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:16:03.660219 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.660135 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d608241e-e3ab-4d43-bd7c-b80481b5d9f7" (UID: "d608241e-e3ab-4d43-bd7c-b80481b5d9f7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:16:03.660306 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.660231 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d608241e-e3ab-4d43-bd7c-b80481b5d9f7" (UID: "d608241e-e3ab-4d43-bd7c-b80481b5d9f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:16:03.661989 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.661968 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d608241e-e3ab-4d43-bd7c-b80481b5d9f7" (UID: "d608241e-e3ab-4d43-bd7c-b80481b5d9f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:16:03.662088 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.662023 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d608241e-e3ab-4d43-bd7c-b80481b5d9f7" (UID: "d608241e-e3ab-4d43-bd7c-b80481b5d9f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:16:03.760690 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.760651 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hrhb\" (UniqueName: \"kubernetes.io/projected/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-kube-api-access-6hrhb\") pod \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\" (UID: \"d608241e-e3ab-4d43-bd7c-b80481b5d9f7\") " Jul 28 10:16:03.760888 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.760822 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:16:03.760888 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.760834 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-service-ca\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:16:03.760888 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.760842 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-trusted-ca-bundle\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:16:03.760888 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.760852 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-oauth-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:16:03.760888 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.760860 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-oauth-serving-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:16:03.760888 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.760869 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-console-serving-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:16:03.762723 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.762695 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-kube-api-access-6hrhb" (OuterVolumeSpecName: "kube-api-access-6hrhb") pod "d608241e-e3ab-4d43-bd7c-b80481b5d9f7" (UID: "d608241e-e3ab-4d43-bd7c-b80481b5d9f7"). InnerVolumeSpecName "kube-api-access-6hrhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:16:03.861260 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:03.861179 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrhb\" (UniqueName: \"kubernetes.io/projected/d608241e-e3ab-4d43-bd7c-b80481b5d9f7-kube-api-access-6hrhb\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:16:04.332457 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:04.332426 2569 generic.go:356] "Generic (PLEG): container finished" podID="d608241e-e3ab-4d43-bd7c-b80481b5d9f7" containerID="10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4" exitCode=0 Jul 28 10:16:04.332939 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:04.332472 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78c8cdcf-rwj92" Jul 28 10:16:04.332939 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:04.332476 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78c8cdcf-rwj92" event={"ID":"d608241e-e3ab-4d43-bd7c-b80481b5d9f7","Type":"ContainerDied","Data":"10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4"} Jul 28 10:16:04.332939 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:04.332517 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78c8cdcf-rwj92" event={"ID":"d608241e-e3ab-4d43-bd7c-b80481b5d9f7","Type":"ContainerDied","Data":"938563f06c3dc698397cffe3ccb822cb4d33e78cd13bf1ffe2e311de9655159c"} Jul 28 10:16:04.332939 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:04.332536 2569 scope.go:117] "RemoveContainer" containerID="10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4" Jul 28 10:16:04.340708 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:04.340689 2569 scope.go:117] "RemoveContainer" containerID="10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4" Jul 28 10:16:04.340993 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:16:04.340968 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4\": container with ID starting with 10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4 not found: ID does not exist" containerID="10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4" Jul 28 10:16:04.341087 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:04.340999 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4"} err="failed to get container status \"10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4\": rpc error: code = NotFound desc = could not find container \"10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4\": container with ID starting with 10285dfa835a105c7b3f474344000a2d524cbc20bc837ea7b3c9bdb95e7994d4 not found: ID does not exist" Jul 28 10:16:04.352386 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:04.352364 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b78c8cdcf-rwj92"] Jul 28 10:16:04.353761 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:04.353740 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b78c8cdcf-rwj92"] Jul 28 10:16:06.079105 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:06.079071 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d608241e-e3ab-4d43-bd7c-b80481b5d9f7" path="/var/lib/kubelet/pods/d608241e-e3ab-4d43-bd7c-b80481b5d9f7/volumes" Jul 28 10:16:09.382983 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.382948 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6968749498-l45v7"] Jul 28 10:16:09.383395 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.383153 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6968749498-l45v7" podUID="1e3ad91d-6aac-4486-944c-e7a9e1e57708" containerName="manager" containerID="cri-o://a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667" gracePeriod=10 Jul 28 10:16:09.406261 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.406232 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6968749498-5zm2t"] Jul 28 10:16:09.406605 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.406590 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d608241e-e3ab-4d43-bd7c-b80481b5d9f7" containerName="console" Jul 28 10:16:09.406605 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.406604 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d608241e-e3ab-4d43-bd7c-b80481b5d9f7" containerName="console" Jul 28 10:16:09.406754 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.406676 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d608241e-e3ab-4d43-bd7c-b80481b5d9f7" containerName="console" Jul 28 10:16:09.408514 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.408494 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:09.417193 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.417165 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-5zm2t"] Jul 28 10:16:09.498334 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.498311 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfkgg\" (UniqueName: \"kubernetes.io/projected/2dd44e98-7fd3-48f0-acd0-d409409bcad7-kube-api-access-wfkgg\") pod \"kserve-controller-manager-6968749498-5zm2t\" (UID: \"2dd44e98-7fd3-48f0-acd0-d409409bcad7\") " pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:09.498448 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.498433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dd44e98-7fd3-48f0-acd0-d409409bcad7-cert\") pod \"kserve-controller-manager-6968749498-5zm2t\" (UID: \"2dd44e98-7fd3-48f0-acd0-d409409bcad7\") " pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:09.598871 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.598842 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dd44e98-7fd3-48f0-acd0-d409409bcad7-cert\") pod \"kserve-controller-manager-6968749498-5zm2t\" (UID: \"2dd44e98-7fd3-48f0-acd0-d409409bcad7\") " pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:09.599025 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.598881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfkgg\" (UniqueName: \"kubernetes.io/projected/2dd44e98-7fd3-48f0-acd0-d409409bcad7-kube-api-access-wfkgg\") pod \"kserve-controller-manager-6968749498-5zm2t\" (UID: \"2dd44e98-7fd3-48f0-acd0-d409409bcad7\") " pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:09.601002 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.600975 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dd44e98-7fd3-48f0-acd0-d409409bcad7-cert\") pod \"kserve-controller-manager-6968749498-5zm2t\" (UID: \"2dd44e98-7fd3-48f0-acd0-d409409bcad7\") " pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:09.608095 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.608073 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfkgg\" (UniqueName: \"kubernetes.io/projected/2dd44e98-7fd3-48f0-acd0-d409409bcad7-kube-api-access-wfkgg\") pod \"kserve-controller-manager-6968749498-5zm2t\" (UID: \"2dd44e98-7fd3-48f0-acd0-d409409bcad7\") " pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:09.627189 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.627169 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:16:09.699739 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.699670 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3ad91d-6aac-4486-944c-e7a9e1e57708-cert\") pod \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\" (UID: \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\") " Jul 28 10:16:09.699739 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.699718 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nwmd\" (UniqueName: \"kubernetes.io/projected/1e3ad91d-6aac-4486-944c-e7a9e1e57708-kube-api-access-4nwmd\") pod \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\" (UID: \"1e3ad91d-6aac-4486-944c-e7a9e1e57708\") " Jul 28 10:16:09.701662 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.701638 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3ad91d-6aac-4486-944c-e7a9e1e57708-cert" (OuterVolumeSpecName: "cert") pod "1e3ad91d-6aac-4486-944c-e7a9e1e57708" (UID: "1e3ad91d-6aac-4486-944c-e7a9e1e57708"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:16:09.701762 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.701741 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3ad91d-6aac-4486-944c-e7a9e1e57708-kube-api-access-4nwmd" (OuterVolumeSpecName: "kube-api-access-4nwmd") pod "1e3ad91d-6aac-4486-944c-e7a9e1e57708" (UID: "1e3ad91d-6aac-4486-944c-e7a9e1e57708"). InnerVolumeSpecName "kube-api-access-4nwmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:16:09.758578 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.758555 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:09.800610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.800582 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nwmd\" (UniqueName: \"kubernetes.io/projected/1e3ad91d-6aac-4486-944c-e7a9e1e57708-kube-api-access-4nwmd\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:16:09.800610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.800610 2569 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3ad91d-6aac-4486-944c-e7a9e1e57708-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:16:09.887713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:09.887689 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-5zm2t"] Jul 28 10:16:09.890296 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:16:09.890270 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd44e98_7fd3_48f0_acd0_d409409bcad7.slice/crio-8d5cb0c660b410df31da8999ceccce98dba9707a58c15829cb0901b41313d809 WatchSource:0}: Error finding container 8d5cb0c660b410df31da8999ceccce98dba9707a58c15829cb0901b41313d809: Status 404 returned error can't find the container with id 8d5cb0c660b410df31da8999ceccce98dba9707a58c15829cb0901b41313d809 Jul 28 10:16:10.358267 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.358236 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-5zm2t" event={"ID":"2dd44e98-7fd3-48f0-acd0-d409409bcad7","Type":"ContainerStarted","Data":"8d5cb0c660b410df31da8999ceccce98dba9707a58c15829cb0901b41313d809"} Jul 28 10:16:10.359403 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.359382 2569 generic.go:356] "Generic (PLEG): container finished" podID="1e3ad91d-6aac-4486-944c-e7a9e1e57708" containerID="a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667" exitCode=0 Jul 28 10:16:10.359504 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.359427 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-l45v7" Jul 28 10:16:10.359561 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.359417 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-l45v7" event={"ID":"1e3ad91d-6aac-4486-944c-e7a9e1e57708","Type":"ContainerDied","Data":"a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667"} Jul 28 10:16:10.359689 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.359559 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-l45v7" event={"ID":"1e3ad91d-6aac-4486-944c-e7a9e1e57708","Type":"ContainerDied","Data":"e99ffd2cf0ed257f9fa34405c9b5a239a52bbcf5e2bb3097f66cb27e5c6a72d0"} Jul 28 10:16:10.359689 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.359593 2569 scope.go:117] "RemoveContainer" containerID="a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667" Jul 28 10:16:10.367325 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.367297 2569 scope.go:117] "RemoveContainer" containerID="a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667" Jul 28 10:16:10.367588 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:16:10.367569 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667\": container with ID starting with a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667 not found: ID does not exist" containerID="a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667" Jul 28 10:16:10.367666 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.367596 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667"} err="failed to get container status \"a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667\": rpc error: code = NotFound desc = could not find container \"a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667\": container with ID starting with a1b962732adca334f7430b542d9b9ebac6cc0cfb6ff04cce00d4c1cafd7a3667 not found: ID does not exist" Jul 28 10:16:10.378435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.378404 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6968749498-l45v7"] Jul 28 10:16:10.384029 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:10.383999 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6968749498-l45v7"] Jul 28 10:16:11.365451 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:11.365417 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-5zm2t" event={"ID":"2dd44e98-7fd3-48f0-acd0-d409409bcad7","Type":"ContainerStarted","Data":"d7380129a165b047ffe99c61ecec45515bd29cba836d7b5792916b6f9560bde6"} Jul 28 10:16:11.365667 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:11.365521 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:11.383636 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:11.383572 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6968749498-5zm2t" podStartSLOduration=1.9676164059999999 podStartE2EDuration="2.383555475s" totalImagesPullingTime="415.939069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:16:11.383238958 +0000 UTC m=+515.825817932" watchObservedRunningTime="2026-07-28 10:16:11.383555475 +0000 UTC m=+515.826134451" Jul 28 10:16:12.078402 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:12.078362 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e3ad91d-6aac-4486-944c-e7a9e1e57708" path="/var/lib/kubelet/pods/1e3ad91d-6aac-4486-944c-e7a9e1e57708/volumes" Jul 28 10:16:42.375921 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:42.375887 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6968749498-5zm2t" Jul 28 10:16:43.410543 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.410506 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-djbtd"] Jul 28 10:16:43.410942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.410846 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3ad91d-6aac-4486-944c-e7a9e1e57708" containerName="manager" Jul 28 10:16:43.410942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.410857 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3ad91d-6aac-4486-944c-e7a9e1e57708" containerName="manager" Jul 28 10:16:43.410942 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.410937 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3ad91d-6aac-4486-944c-e7a9e1e57708" containerName="manager" Jul 28 10:16:43.415331 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.415311 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:43.418578 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.418557 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 28 10:16:43.418696 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.418649 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-9wpvp\"" Jul 28 10:16:43.436561 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.436530 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-djbtd"] Jul 28 10:16:43.453271 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.453239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9-cert\") pod \"odh-model-controller-75d9cdbddf-djbtd\" (UID: \"9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9\") " pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:43.453441 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.453283 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4l4d\" (UniqueName: \"kubernetes.io/projected/9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9-kube-api-access-s4l4d\") pod \"odh-model-controller-75d9cdbddf-djbtd\" (UID: \"9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9\") " pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:43.554738 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.554697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9-cert\") pod \"odh-model-controller-75d9cdbddf-djbtd\" (UID: \"9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9\") " pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:43.554738 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.554738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4l4d\" (UniqueName: \"kubernetes.io/projected/9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9-kube-api-access-s4l4d\") pod \"odh-model-controller-75d9cdbddf-djbtd\" (UID: \"9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9\") " pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:43.554966 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:16:43.554846 2569 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 28 10:16:43.554966 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:16:43.554913 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9-cert podName:9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9 nodeName:}" failed. No retries permitted until 2026-07-28 10:16:44.054897594 +0000 UTC m=+548.497476551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9-cert") pod "odh-model-controller-75d9cdbddf-djbtd" (UID: "9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9") : secret "odh-model-controller-webhook-cert" not found Jul 28 10:16:43.565968 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:43.565937 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4l4d\" (UniqueName: \"kubernetes.io/projected/9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9-kube-api-access-s4l4d\") pod \"odh-model-controller-75d9cdbddf-djbtd\" (UID: \"9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9\") " pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:44.059534 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:44.059495 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9-cert\") pod \"odh-model-controller-75d9cdbddf-djbtd\" (UID: \"9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9\") " pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:44.061776 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:44.061754 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9-cert\") pod \"odh-model-controller-75d9cdbddf-djbtd\" (UID: \"9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9\") " pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:44.325342 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:44.325259 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:44.452405 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:44.452378 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-djbtd"] Jul 28 10:16:44.454857 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:16:44.454834 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4c9c7b_9f2b_4719_9bb9_ad57f9d195f9.slice/crio-e5881631f5f3bbde5b07dc9d4e1c2a57fb9fc55de9d6f7efaff3c45250770362 WatchSource:0}: Error finding container e5881631f5f3bbde5b07dc9d4e1c2a57fb9fc55de9d6f7efaff3c45250770362: Status 404 returned error can't find the container with id e5881631f5f3bbde5b07dc9d4e1c2a57fb9fc55de9d6f7efaff3c45250770362 Jul 28 10:16:44.505530 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:44.505498 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-djbtd" event={"ID":"9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9","Type":"ContainerStarted","Data":"e5881631f5f3bbde5b07dc9d4e1c2a57fb9fc55de9d6f7efaff3c45250770362"} Jul 28 10:16:47.520950 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:47.520912 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-djbtd" event={"ID":"9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9","Type":"ContainerStarted","Data":"4a73db18366d5178780558822b3f8c12b9b8f960ce25e3daacdead91983b551e"} Jul 28 10:16:47.521416 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:47.521073 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:16:47.538764 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:47.538721 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-djbtd" podStartSLOduration=1.6752598509999999 podStartE2EDuration="4.53870652s" totalImagesPullingTime="2.863446669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:16:47.537748118 +0000 UTC m=+551.980327104" watchObservedRunningTime="2026-07-28 10:16:47.53870652 +0000 UTC m=+551.981285495" Jul 28 10:16:58.527888 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:16:58.527855 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-djbtd" Jul 28 10:17:35.946059 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:17:35.946032 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:17:35.949557 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:17:35.949531 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:21:53.417859 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.417827 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw"] Jul 28 10:21:53.421442 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.421426 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:21:53.425455 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.425434 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 10:21:53.426571 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.426552 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-74e2d-serving-cert\"" Jul 28 10:21:53.426828 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.426812 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d7gr5\"" Jul 28 10:21:53.427141 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.427123 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-74e2d-kube-rbac-proxy-sar-config\"" Jul 28 10:21:53.440949 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.440923 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw"] Jul 28 10:21:53.458329 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.458307 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8220dfe-dd2a-4539-b3ba-d259965a65c8-proxy-tls\") pod \"model-chainer-raw-74e2d-864c794844-n9cgw\" (UID: \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\") " pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:21:53.458464 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.458375 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8220dfe-dd2a-4539-b3ba-d259965a65c8-openshift-service-ca-bundle\") pod \"model-chainer-raw-74e2d-864c794844-n9cgw\" (UID: \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\") " pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:21:53.559562 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.559528 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8220dfe-dd2a-4539-b3ba-d259965a65c8-proxy-tls\") pod \"model-chainer-raw-74e2d-864c794844-n9cgw\" (UID: \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\") " pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:21:53.559765 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.559584 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8220dfe-dd2a-4539-b3ba-d259965a65c8-openshift-service-ca-bundle\") pod \"model-chainer-raw-74e2d-864c794844-n9cgw\" (UID: \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\") " pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:21:53.560241 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.560223 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8220dfe-dd2a-4539-b3ba-d259965a65c8-openshift-service-ca-bundle\") pod \"model-chainer-raw-74e2d-864c794844-n9cgw\" (UID: \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\") " pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:21:53.561959 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.561939 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8220dfe-dd2a-4539-b3ba-d259965a65c8-proxy-tls\") pod \"model-chainer-raw-74e2d-864c794844-n9cgw\" (UID: \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\") " pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:21:53.731546 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.731469 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:21:53.861397 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.861345 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw"] Jul 28 10:21:53.864295 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:21:53.864263 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8220dfe_dd2a_4539_b3ba_d259965a65c8.slice/crio-e02ca1609f0bd2d3447ddd977060006b0c08711d15781c2c0474fb5a4ca9c034 WatchSource:0}: Error finding container e02ca1609f0bd2d3447ddd977060006b0c08711d15781c2c0474fb5a4ca9c034: Status 404 returned error can't find the container with id e02ca1609f0bd2d3447ddd977060006b0c08711d15781c2c0474fb5a4ca9c034 Jul 28 10:21:53.866807 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:53.866788 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 10:21:54.776639 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:54.776585 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" event={"ID":"a8220dfe-dd2a-4539-b3ba-d259965a65c8","Type":"ContainerStarted","Data":"e02ca1609f0bd2d3447ddd977060006b0c08711d15781c2c0474fb5a4ca9c034"} Jul 28 10:21:56.787629 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:56.787583 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" event={"ID":"a8220dfe-dd2a-4539-b3ba-d259965a65c8","Type":"ContainerStarted","Data":"c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc"} Jul 28 10:21:56.788001 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:56.787717 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:21:56.805054 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:21:56.805015 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" podStartSLOduration=1.706272533 podStartE2EDuration="3.805000528s" totalImagesPullingTime="2.098727995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:21:56.804185513 +0000 UTC m=+861.246764487" watchObservedRunningTime="2026-07-28 10:21:56.805000528 +0000 UTC m=+861.247579535" Jul 28 10:22:02.798399 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:02.798368 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:22:03.394509 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:03.394475 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw"] Jul 28 10:22:03.394747 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:03.394715 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" containerID="cri-o://c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc" gracePeriod=30 Jul 28 10:22:07.796115 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:07.796069 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:22:12.796341 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:12.796294 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:22:17.796353 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:17.796316 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:22:17.796783 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:17.796420 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:22:22.796748 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:22.796699 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:22:27.796521 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:27.796434 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:22:32.796528 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:32.796486 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:22:33.542999 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.542977 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:22:33.695996 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.695910 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8220dfe-dd2a-4539-b3ba-d259965a65c8-proxy-tls\") pod \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\" (UID: \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\") " Jul 28 10:22:33.696145 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.695998 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8220dfe-dd2a-4539-b3ba-d259965a65c8-openshift-service-ca-bundle\") pod \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\" (UID: \"a8220dfe-dd2a-4539-b3ba-d259965a65c8\") " Jul 28 10:22:33.696352 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.696318 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8220dfe-dd2a-4539-b3ba-d259965a65c8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a8220dfe-dd2a-4539-b3ba-d259965a65c8" (UID: "a8220dfe-dd2a-4539-b3ba-d259965a65c8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:22:33.697992 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.697973 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8220dfe-dd2a-4539-b3ba-d259965a65c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8220dfe-dd2a-4539-b3ba-d259965a65c8" (UID: "a8220dfe-dd2a-4539-b3ba-d259965a65c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:22:33.797023 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.796995 2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8220dfe-dd2a-4539-b3ba-d259965a65c8-proxy-tls\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:22:33.797023 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.797018 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8220dfe-dd2a-4539-b3ba-d259965a65c8-openshift-service-ca-bundle\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:22:33.952292 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.952195 2569 generic.go:356] "Generic (PLEG): container finished" podID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerID="c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc" exitCode=0 Jul 28 10:22:33.952292 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.952239 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" Jul 28 10:22:33.952578 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.952300 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" event={"ID":"a8220dfe-dd2a-4539-b3ba-d259965a65c8","Type":"ContainerDied","Data":"c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc"} Jul 28 10:22:33.952578 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.952353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw" event={"ID":"a8220dfe-dd2a-4539-b3ba-d259965a65c8","Type":"ContainerDied","Data":"e02ca1609f0bd2d3447ddd977060006b0c08711d15781c2c0474fb5a4ca9c034"} Jul 28 10:22:33.952578 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.952375 2569 scope.go:117] "RemoveContainer" containerID="c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc" Jul 28 10:22:33.961512 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.961497 2569 scope.go:117] "RemoveContainer" containerID="c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc" Jul 28 10:22:33.961769 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:22:33.961747 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc\": container with ID starting with c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc not found: ID does not exist" containerID="c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc" Jul 28 10:22:33.961842 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.961782 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc"} err="failed to get container status \"c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc\": rpc error: code = NotFound desc = could not find container \"c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc\": container with ID starting with c1a414e274af663a7b1de3cc316896966c97405de58a10efa6be04627e2b4edc not found: ID does not exist" Jul 28 10:22:33.975498 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.975476 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw"] Jul 28 10:22:33.977452 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:33.977432 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-74e2d-864c794844-n9cgw"] Jul 28 10:22:34.078894 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:34.078867 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" path="/var/lib/kubelet/pods/a8220dfe-dd2a-4539-b3ba-d259965a65c8/volumes" Jul 28 10:22:35.972782 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:35.972754 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:22:35.974217 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:22:35.974195 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:23:43.677601 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.677566 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9"] Jul 28 10:23:43.678060 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.677946 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" Jul 28 10:23:43.678060 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.677959 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" Jul 28 10:23:43.678060 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.678014 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8220dfe-dd2a-4539-b3ba-d259965a65c8" containerName="model-chainer-raw-74e2d" Jul 28 10:23:43.680769 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.680748 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:43.683871 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.683850 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-b0501-serving-cert\"" Jul 28 10:23:43.683972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.683935 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d7gr5\"" Jul 28 10:23:43.684960 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.684939 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-b0501-kube-rbac-proxy-sar-config\"" Jul 28 10:23:43.685055 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.684946 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 10:23:43.693745 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.693724 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9"] Jul 28 10:23:43.751327 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.751296 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-proxy-tls\") pod \"model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9\" (UID: \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:43.751467 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.751343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9\" (UID: \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:43.852405 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.852372 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-proxy-tls\") pod \"model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9\" (UID: \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:43.852552 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.852422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9\" (UID: \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:43.852552 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:23:43.852533 2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-serving-cert: secret "model-chainer-raw-hpa-b0501-serving-cert" not found Jul 28 10:23:43.852667 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:23:43.852636 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-proxy-tls podName:c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc nodeName:}" failed. No retries permitted until 2026-07-28 10:23:44.35259695 +0000 UTC m=+968.795175906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-proxy-tls") pod "model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" (UID: "c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc") : secret "model-chainer-raw-hpa-b0501-serving-cert" not found Jul 28 10:23:43.853021 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:43.853004 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9\" (UID: \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:44.357972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:44.357919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-proxy-tls\") pod \"model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9\" (UID: \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:44.360290 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:44.360272 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-proxy-tls\") pod \"model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9\" (UID: \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:44.590481 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:44.590447 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:44.716726 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:44.716692 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9"] Jul 28 10:23:44.720149 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:23:44.720122 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58e9c1c_f8f7_447e_b7f7_9b6a215d16cc.slice/crio-137e78f3c110c5d9e85a0bb0b1c91e039748c3cd48c53a12a06bbbb334f7f5b1 WatchSource:0}: Error finding container 137e78f3c110c5d9e85a0bb0b1c91e039748c3cd48c53a12a06bbbb334f7f5b1: Status 404 returned error can't find the container with id 137e78f3c110c5d9e85a0bb0b1c91e039748c3cd48c53a12a06bbbb334f7f5b1 Jul 28 10:23:45.232211 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:45.232176 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" event={"ID":"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc","Type":"ContainerStarted","Data":"672ff98d612b7793fed7ac894e274cf0d8754b995e9196dcb30350440fe30047"} Jul 28 10:23:45.232379 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:45.232229 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" event={"ID":"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc","Type":"ContainerStarted","Data":"137e78f3c110c5d9e85a0bb0b1c91e039748c3cd48c53a12a06bbbb334f7f5b1"} Jul 28 10:23:45.232379 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:45.232257 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:45.250122 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:45.250076 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" podStartSLOduration=2.250059387 podStartE2EDuration="2.250059387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:45.248819723 +0000 UTC m=+969.691398699" watchObservedRunningTime="2026-07-28 10:23:45.250059387 +0000 UTC m=+969.692638361" Jul 28 10:23:51.242401 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:51.242372 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:23:53.769728 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:53.769684 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9"] Jul 28 10:23:53.770139 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:53.769911 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" containerID="cri-o://672ff98d612b7793fed7ac894e274cf0d8754b995e9196dcb30350440fe30047" gracePeriod=30 Jul 28 10:23:56.240758 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:23:56.240722 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:24:01.240640 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:01.240538 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:24:06.240730 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:06.240690 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:24:06.241187 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:06.240826 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:24:11.240035 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:11.239996 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:24:16.240901 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:16.240863 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:24:21.240388 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:21.240342 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 10:24:24.398932 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:24.398897 2569 generic.go:356] "Generic (PLEG): container finished" podID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerID="672ff98d612b7793fed7ac894e274cf0d8754b995e9196dcb30350440fe30047" exitCode=0 Jul 28 10:24:24.399348 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:24.398973 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" event={"ID":"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc","Type":"ContainerDied","Data":"672ff98d612b7793fed7ac894e274cf0d8754b995e9196dcb30350440fe30047"} Jul 28 10:24:24.420824 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:24.420801 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:24:24.484242 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:24.484217 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-proxy-tls\") pod \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\" (UID: \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\") " Jul 28 10:24:24.484391 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:24.484249 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-openshift-service-ca-bundle\") pod \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\" (UID: \"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc\") " Jul 28 10:24:24.484650 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:24.484608 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" (UID: "c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:24:24.486155 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:24.486133 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" (UID: "c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:24:24.585242 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:24.585170 2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-proxy-tls\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:24:24.585242 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:24.585197 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc-openshift-service-ca-bundle\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:24:25.410127 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:25.410089 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" event={"ID":"c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc","Type":"ContainerDied","Data":"137e78f3c110c5d9e85a0bb0b1c91e039748c3cd48c53a12a06bbbb334f7f5b1"} Jul 28 10:24:25.410607 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:25.410173 2569 scope.go:117] "RemoveContainer" containerID="672ff98d612b7793fed7ac894e274cf0d8754b995e9196dcb30350440fe30047" Jul 28 10:24:25.410607 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:25.410102 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9" Jul 28 10:24:25.431517 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:25.431493 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9"] Jul 28 10:24:25.434916 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:25.434896 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-b0501-578b7cf5fb-xbwt9"] Jul 28 10:24:26.078729 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:24:26.078695 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" path="/var/lib/kubelet/pods/c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc/volumes" Jul 28 10:27:36.000028 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:27:35.999957 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:27:36.000028 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:27:36.000015 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:31:40.065074 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.064987 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p"] Jul 28 10:31:40.065553 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.065342 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" Jul 28 10:31:40.065553 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.065354 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" Jul 28 10:31:40.065553 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.065415 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c58e9c1c-f8f7-447e-b7f7-9b6a215d16cc" containerName="model-chainer-raw-hpa-b0501" Jul 28 10:31:40.068437 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.068420 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.071490 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.071467 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Jul 28 10:31:40.071490 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.071467 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 28 10:31:40.071727 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.071522 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 28 10:31:40.071727 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.071719 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 28 10:31:40.072716 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.072696 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Jul 28 10:31:40.072831 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.072728 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 28 10:31:40.074259 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.074242 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d7gr5\"" Jul 28 10:31:40.087872 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.087845 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p"] Jul 28 10:31:40.104944 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.104919 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgdbd\" (UniqueName: \"kubernetes.io/projected/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-api-access-kgdbd\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.105068 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.105026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.105135 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.105063 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.105135 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.105100 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.105225 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.105138 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.206351 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.206318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.206351 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.206351 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.206600 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.206373 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.206600 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.206391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.206600 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.206454 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgdbd\" (UniqueName: \"kubernetes.io/projected/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-api-access-kgdbd\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.206834 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.206811 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.207243 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.207223 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.207278 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.207223 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.208771 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.208755 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.219391 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.219369 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgdbd\" (UniqueName: \"kubernetes.io/projected/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-api-access-kgdbd\") pod \"isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.380198 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.380100 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:31:40.514612 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.514587 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p"] Jul 28 10:31:40.516800 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:31:40.516764 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6735ba4f_8d04_4b99_91e2_ae74fa3243a7.slice/crio-fd8135a573be9d82eed850c154c5ee08ff1ac8ee6bfa96e0486596a4979bb6ed WatchSource:0}: Error finding container fd8135a573be9d82eed850c154c5ee08ff1ac8ee6bfa96e0486596a4979bb6ed: Status 404 returned error can't find the container with id fd8135a573be9d82eed850c154c5ee08ff1ac8ee6bfa96e0486596a4979bb6ed Jul 28 10:31:40.518825 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:40.518804 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 10:31:41.188339 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:41.188295 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" event={"ID":"6735ba4f-8d04-4b99-91e2-ae74fa3243a7","Type":"ContainerStarted","Data":"fd8135a573be9d82eed850c154c5ee08ff1ac8ee6bfa96e0486596a4979bb6ed"} Jul 28 10:31:44.205491 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:44.205451 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" event={"ID":"6735ba4f-8d04-4b99-91e2-ae74fa3243a7","Type":"ContainerStarted","Data":"84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2"} Jul 28 10:31:48.224892 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:48.224855 2569 generic.go:356] "Generic (PLEG): container finished" podID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerID="84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2" exitCode=0 Jul 28 10:31:48.225258 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:31:48.224927 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" event={"ID":"6735ba4f-8d04-4b99-91e2-ae74fa3243a7","Type":"ContainerDied","Data":"84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2"} Jul 28 10:32:02.302211 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:02.302172 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" event={"ID":"6735ba4f-8d04-4b99-91e2-ae74fa3243a7","Type":"ContainerStarted","Data":"dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f"} Jul 28 10:32:04.313904 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:04.313868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" event={"ID":"6735ba4f-8d04-4b99-91e2-ae74fa3243a7","Type":"ContainerStarted","Data":"be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877"} Jul 28 10:32:04.314363 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:04.313973 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:32:04.333803 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:04.333751 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podStartSLOduration=1.316589116 podStartE2EDuration="24.333737316s" totalImagesPullingTime="19.069364681s" totalInitContainerRuntime="3.947783519s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:31:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:32:04.333083879 +0000 UTC m=+1468.775662853" watchObservedRunningTime="2026-07-28 10:32:04.333737316 +0000 UTC m=+1468.776316290" Jul 28 10:32:05.320207 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:05.320173 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:32:05.321532 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:05.321503 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:32:06.324401 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:06.324353 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:32:11.329010 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:11.328981 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:32:11.329593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:11.329566 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:32:21.330045 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:21.330000 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:32:31.330050 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:31.330009 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:32:36.027425 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:36.027395 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:32:36.027913 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:36.027496 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:32:41.330317 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:41.330280 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:32:51.330186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:32:51.330143 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:33:01.329519 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:01.329427 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:33:11.329670 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:11.329603 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:33:21.330810 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:21.330775 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:33:30.221079 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:30.221040 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p"] Jul 28 10:33:30.221674 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:30.221329 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" containerID="cri-o://dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f" gracePeriod=30 Jul 28 10:33:30.221674 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:30.221541 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kube-rbac-proxy" containerID="cri-o://be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877" gracePeriod=30 Jul 28 10:33:30.696836 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:30.696800 2569 generic.go:356] "Generic (PLEG): container finished" podID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerID="be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877" exitCode=2 Jul 28 10:33:30.697006 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:30.696841 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" event={"ID":"6735ba4f-8d04-4b99-91e2-ae74fa3243a7","Type":"ContainerDied","Data":"be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877"} Jul 28 10:33:31.324986 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:31.324943 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused" Jul 28 10:33:31.330322 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:31.330298 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Jul 28 10:33:34.874114 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.874090 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:33:34.979387 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.979314 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgdbd\" (UniqueName: \"kubernetes.io/projected/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-api-access-kgdbd\") pod \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " Jul 28 10:33:34.979387 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.979359 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-rbac-proxy-sar-config\") pod \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " Jul 28 10:33:34.979560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.979400 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kserve-provision-location\") pod \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " Jul 28 10:33:34.979560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.979447 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-proxy-tls\") pod \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " Jul 28 10:33:34.979560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.979509 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-cabundle-cert\") pod \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\" (UID: \"6735ba4f-8d04-4b99-91e2-ae74fa3243a7\") " Jul 28 10:33:34.979804 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.979760 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6735ba4f-8d04-4b99-91e2-ae74fa3243a7" (UID: "6735ba4f-8d04-4b99-91e2-ae74fa3243a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:33:34.979952 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.979779 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6735ba4f-8d04-4b99-91e2-ae74fa3243a7" (UID: "6735ba4f-8d04-4b99-91e2-ae74fa3243a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:33:34.979952 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.979858 2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:33:34.979952 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.979926 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6735ba4f-8d04-4b99-91e2-ae74fa3243a7" (UID: "6735ba4f-8d04-4b99-91e2-ae74fa3243a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:33:34.981372 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.981341 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-api-access-kgdbd" (OuterVolumeSpecName: "kube-api-access-kgdbd") pod "6735ba4f-8d04-4b99-91e2-ae74fa3243a7" (UID: "6735ba4f-8d04-4b99-91e2-ae74fa3243a7"). InnerVolumeSpecName "kube-api-access-kgdbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:33:34.981448 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:34.981425 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6735ba4f-8d04-4b99-91e2-ae74fa3243a7" (UID: "6735ba4f-8d04-4b99-91e2-ae74fa3243a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:33:35.080758 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.080714 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgdbd\" (UniqueName: \"kubernetes.io/projected/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kube-api-access-kgdbd\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:33:35.080758 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.080753 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-kserve-provision-location\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:33:35.080955 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.080784 2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-proxy-tls\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:33:35.080955 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.080793 2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6735ba4f-8d04-4b99-91e2-ae74fa3243a7-cabundle-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:33:35.389805 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.389772 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk"] Jul 28 10:33:35.390135 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.390122 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="storage-initializer" Jul 28 10:33:35.390186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.390136 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="storage-initializer" Jul 28 10:33:35.390186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.390151 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kube-rbac-proxy" Jul 28 10:33:35.390186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.390157 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kube-rbac-proxy" Jul 28 10:33:35.390186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.390168 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" Jul 28 10:33:35.390186 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.390173 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" Jul 28 10:33:35.390349 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.390222 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kube-rbac-proxy" Jul 28 10:33:35.390349 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.390234 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerName="kserve-container" Jul 28 10:33:35.393509 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.393483 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.396257 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.396233 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\"" Jul 28 10:33:35.396257 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.396253 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Jul 28 10:33:35.405684 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.405659 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk"] Jul 28 10:33:35.485036 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.485003 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82lcn\" (UniqueName: \"kubernetes.io/projected/820a319b-613b-4eea-b074-506589384b55-kube-api-access-82lcn\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.485189 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.485043 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.485189 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.485087 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/820a319b-613b-4eea-b074-506589384b55-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.485189 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.485153 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.485189 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.485174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820a319b-613b-4eea-b074-506589384b55-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.586404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.586355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82lcn\" (UniqueName: \"kubernetes.io/projected/820a319b-613b-4eea-b074-506589384b55-kube-api-access-82lcn\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.586404 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.586408 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.586702 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.586440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/820a319b-613b-4eea-b074-506589384b55-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.586702 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.586475 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.586702 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.586490 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820a319b-613b-4eea-b074-506589384b55-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.586702 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:33:35.586591 2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found Jul 28 10:33:35.586702 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:33:35.586683 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/820a319b-613b-4eea-b074-506589384b55-proxy-tls podName:820a319b-613b-4eea-b074-506589384b55 nodeName:}" failed. No retries permitted until 2026-07-28 10:33:36.086662882 +0000 UTC m=+1560.529241836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/820a319b-613b-4eea-b074-506589384b55-proxy-tls") pod "isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" (UID: "820a319b-613b-4eea-b074-506589384b55") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found Jul 28 10:33:35.586926 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.586909 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820a319b-613b-4eea-b074-506589384b55-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.587227 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.587209 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.587263 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.587210 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.596151 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.596129 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82lcn\" (UniqueName: \"kubernetes.io/projected/820a319b-613b-4eea-b074-506589384b55-kube-api-access-82lcn\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:35.721593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.721504 2569 generic.go:356] "Generic (PLEG): container finished" podID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" containerID="dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f" exitCode=0 Jul 28 10:33:35.721593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.721555 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" Jul 28 10:33:35.721593 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.721579 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" event={"ID":"6735ba4f-8d04-4b99-91e2-ae74fa3243a7","Type":"ContainerDied","Data":"dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f"} Jul 28 10:33:35.721866 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.721652 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p" event={"ID":"6735ba4f-8d04-4b99-91e2-ae74fa3243a7","Type":"ContainerDied","Data":"fd8135a573be9d82eed850c154c5ee08ff1ac8ee6bfa96e0486596a4979bb6ed"} Jul 28 10:33:35.721866 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.721679 2569 scope.go:117] "RemoveContainer" containerID="be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877" Jul 28 10:33:35.730991 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.730976 2569 scope.go:117] "RemoveContainer" containerID="dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f" Jul 28 10:33:35.738254 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.738064 2569 scope.go:117] "RemoveContainer" containerID="84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2" Jul 28 10:33:35.745233 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.745216 2569 scope.go:117] "RemoveContainer" containerID="be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877" Jul 28 10:33:35.745491 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:33:35.745473 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877\": container with ID starting with be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877 not found: ID does not exist" containerID="be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877" Jul 28 10:33:35.745543 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.745499 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877"} err="failed to get container status \"be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877\": rpc error: code = NotFound desc = could not find container \"be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877\": container with ID starting with be0a16e97bd3cce60ec4a24bd99dee3f14164e2d7205c7318f288b7c1c4c3877 not found: ID does not exist" Jul 28 10:33:35.745543 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.745517 2569 scope.go:117] "RemoveContainer" containerID="dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f" Jul 28 10:33:35.745781 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:33:35.745755 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f\": container with ID starting with dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f not found: ID does not exist" containerID="dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f" Jul 28 10:33:35.745835 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.745790 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f"} err="failed to get container status \"dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f\": rpc error: code = NotFound desc = could not find container \"dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f\": container with ID starting with dcfc1361f7d0a579cd7fa99fbb89ee6caf0d3e4d0c36079f7242b268b912967f not found: ID does not exist" Jul 28 10:33:35.745835 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.745811 2569 scope.go:117] "RemoveContainer" containerID="84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2" Jul 28 10:33:35.746074 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:33:35.746025 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2\": container with ID starting with 84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2 not found: ID does not exist" containerID="84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2" Jul 28 10:33:35.746074 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.746045 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2"} err="failed to get container status \"84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2\": rpc error: code = NotFound desc = could not find container \"84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2\": container with ID starting with 84fb2d73abbafa0633b4566b3163323a8577d85b381eeae5db1817bf50e015d2 not found: ID does not exist" Jul 28 10:33:35.746557 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.746539 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p"] Jul 28 10:33:35.751488 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:35.751468 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-74bdc8ffff-b9t5p"] Jul 28 10:33:36.082124 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:36.082089 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6735ba4f-8d04-4b99-91e2-ae74fa3243a7" path="/var/lib/kubelet/pods/6735ba4f-8d04-4b99-91e2-ae74fa3243a7/volumes" Jul 28 10:33:36.091122 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:36.091103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/820a319b-613b-4eea-b074-506589384b55-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:36.093366 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:36.093339 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/820a319b-613b-4eea-b074-506589384b55-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:36.305242 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:36.305209 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:36.439875 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:36.439849 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk"] Jul 28 10:33:36.441947 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:33:36.441916 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820a319b_613b_4eea_b074_506589384b55.slice/crio-1e401dfdd98a0943880f019201a5a9df5637fc6d9f2a8abd04cebf6e5b5b7fac WatchSource:0}: Error finding container 1e401dfdd98a0943880f019201a5a9df5637fc6d9f2a8abd04cebf6e5b5b7fac: Status 404 returned error can't find the container with id 1e401dfdd98a0943880f019201a5a9df5637fc6d9f2a8abd04cebf6e5b5b7fac Jul 28 10:33:36.728211 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:36.728129 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" event={"ID":"820a319b-613b-4eea-b074-506589384b55","Type":"ContainerStarted","Data":"90fa7b17a106f6748387bebc9e38e2b6f6d7a2879a3016a3d196b3cd78b07396"} Jul 28 10:33:36.728211 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:36.728198 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" event={"ID":"820a319b-613b-4eea-b074-506589384b55","Type":"ContainerStarted","Data":"1e401dfdd98a0943880f019201a5a9df5637fc6d9f2a8abd04cebf6e5b5b7fac"} Jul 28 10:33:40.746031 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:40.746000 2569 generic.go:356] "Generic (PLEG): container finished" podID="820a319b-613b-4eea-b074-506589384b55" containerID="90fa7b17a106f6748387bebc9e38e2b6f6d7a2879a3016a3d196b3cd78b07396" exitCode=0 Jul 28 10:33:40.746429 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:40.746037 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" event={"ID":"820a319b-613b-4eea-b074-506589384b55","Type":"ContainerDied","Data":"90fa7b17a106f6748387bebc9e38e2b6f6d7a2879a3016a3d196b3cd78b07396"} Jul 28 10:33:41.752600 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:41.752567 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" event={"ID":"820a319b-613b-4eea-b074-506589384b55","Type":"ContainerStarted","Data":"1cf9cd91000e298eb7b5a2683aecec36219e9249fa3e3ab81498c3ea3ac5b87c"} Jul 28 10:33:41.752972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:41.752642 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" event={"ID":"820a319b-613b-4eea-b074-506589384b55","Type":"ContainerStarted","Data":"eda037efa84acbdcd3910cd85a88252059f58e9a8c8feaa96c6cbcc5e5f8447d"} Jul 28 10:33:41.752972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:41.752831 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:41.775197 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:41.775143 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podStartSLOduration=3.017763098 podStartE2EDuration="6.775126987s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.757363889s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:33:41.773807325 +0000 UTC m=+1566.216386303" watchObservedRunningTime="2026-07-28 10:33:41.775126987 +0000 UTC m=+1566.217706025" Jul 28 10:33:42.756903 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:42.756867 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:42.758111 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:42.758080 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:33:43.761204 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:43.761163 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:33:48.765974 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:48.765943 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:33:48.766457 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:48.766424 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:33:58.767286 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:33:58.767243 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:34:08.767061 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:34:08.767023 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:34:18.766770 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:34:18.766727 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:34:28.766730 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:34:28.766647 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:34:38.766845 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:34:38.766803 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:34:48.766571 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:34:48.766526 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:34:58.767800 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:34:58.767758 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:35:05.414710 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.414675 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk"] Jul 28 10:35:05.415485 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.414913 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" containerID="cri-o://eda037efa84acbdcd3910cd85a88252059f58e9a8c8feaa96c6cbcc5e5f8447d" gracePeriod=30 Jul 28 10:35:05.415485 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.414970 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kube-rbac-proxy" containerID="cri-o://1cf9cd91000e298eb7b5a2683aecec36219e9249fa3e3ab81498c3ea3ac5b87c" gracePeriod=30 Jul 28 10:35:05.489866 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.489823 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q"] Jul 28 10:35:05.493609 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.493577 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.496284 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.496263 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\"" Jul 28 10:35:05.502314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.502291 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q"] Jul 28 10:35:05.539970 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.539934 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded52cbb-42f4-4c5d-98a5-278c2e31547f-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.539970 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.539971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gl5v\" (UniqueName: \"kubernetes.io/projected/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-api-access-4gl5v\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.540157 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.540001 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.540157 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.540074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.540157 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.540104 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.640610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.640567 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.640820 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.640712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.640820 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.640743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.640820 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.640810 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded52cbb-42f4-4c5d-98a5-278c2e31547f-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.641003 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.640838 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gl5v\" (UniqueName: \"kubernetes.io/projected/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-api-access-4gl5v\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.641059 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.641031 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.641560 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.641533 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.641678 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.641534 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.643287 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.643267 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded52cbb-42f4-4c5d-98a5-278c2e31547f-proxy-tls\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.650503 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.650480 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gl5v\" (UniqueName: \"kubernetes.io/projected/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-api-access-4gl5v\") pod \"isvc-canary-rollback-predictor-bb95f9bbd-dzj7q\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.805153 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.805112 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:05.939485 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:05.939455 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q"] Jul 28 10:35:05.942182 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:35:05.942146 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded52cbb_42f4_4c5d_98a5_278c2e31547f.slice/crio-d0d79ed7bfcda8659a38fb5117f1ae3521e4636ff88447ba3fe59f0b4821b2c8 WatchSource:0}: Error finding container d0d79ed7bfcda8659a38fb5117f1ae3521e4636ff88447ba3fe59f0b4821b2c8: Status 404 returned error can't find the container with id d0d79ed7bfcda8659a38fb5117f1ae3521e4636ff88447ba3fe59f0b4821b2c8 Jul 28 10:35:06.126141 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:06.126049 2569 generic.go:356] "Generic (PLEG): container finished" podID="820a319b-613b-4eea-b074-506589384b55" containerID="1cf9cd91000e298eb7b5a2683aecec36219e9249fa3e3ab81498c3ea3ac5b87c" exitCode=2 Jul 28 10:35:06.126141 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:06.126117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" event={"ID":"820a319b-613b-4eea-b074-506589384b55","Type":"ContainerDied","Data":"1cf9cd91000e298eb7b5a2683aecec36219e9249fa3e3ab81498c3ea3ac5b87c"} Jul 28 10:35:06.127713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:06.127689 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" event={"ID":"ded52cbb-42f4-4c5d-98a5-278c2e31547f","Type":"ContainerStarted","Data":"bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713"} Jul 28 10:35:06.127829 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:06.127736 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" event={"ID":"ded52cbb-42f4-4c5d-98a5-278c2e31547f","Type":"ContainerStarted","Data":"d0d79ed7bfcda8659a38fb5117f1ae3521e4636ff88447ba3fe59f0b4821b2c8"} Jul 28 10:35:08.761713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:08.761666 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused" Jul 28 10:35:08.766967 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:08.766931 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Jul 28 10:35:10.150951 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.150915 2569 generic.go:356] "Generic (PLEG): container finished" podID="820a319b-613b-4eea-b074-506589384b55" containerID="eda037efa84acbdcd3910cd85a88252059f58e9a8c8feaa96c6cbcc5e5f8447d" exitCode=0 Jul 28 10:35:10.151337 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.150986 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" event={"ID":"820a319b-613b-4eea-b074-506589384b55","Type":"ContainerDied","Data":"eda037efa84acbdcd3910cd85a88252059f58e9a8c8feaa96c6cbcc5e5f8447d"} Jul 28 10:35:10.152504 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.152484 2569 generic.go:356] "Generic (PLEG): container finished" podID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerID="bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713" exitCode=0 Jul 28 10:35:10.152615 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.152518 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" event={"ID":"ded52cbb-42f4-4c5d-98a5-278c2e31547f","Type":"ContainerDied","Data":"bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713"} Jul 28 10:35:10.164766 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.164750 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:35:10.177597 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.177576 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82lcn\" (UniqueName: \"kubernetes.io/projected/820a319b-613b-4eea-b074-506589384b55-kube-api-access-82lcn\") pod \"820a319b-613b-4eea-b074-506589384b55\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " Jul 28 10:35:10.177716 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.177647 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/820a319b-613b-4eea-b074-506589384b55-proxy-tls\") pod \"820a319b-613b-4eea-b074-506589384b55\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " Jul 28 10:35:10.177774 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.177723 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-kube-rbac-proxy-sar-config\") pod \"820a319b-613b-4eea-b074-506589384b55\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " Jul 28 10:35:10.177774 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.177766 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820a319b-613b-4eea-b074-506589384b55-kserve-provision-location\") pod \"820a319b-613b-4eea-b074-506589384b55\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " Jul 28 10:35:10.177882 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.177795 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-cabundle-cert\") pod \"820a319b-613b-4eea-b074-506589384b55\" (UID: \"820a319b-613b-4eea-b074-506589384b55\") " Jul 28 10:35:10.178464 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.178109 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820a319b-613b-4eea-b074-506589384b55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "820a319b-613b-4eea-b074-506589384b55" (UID: "820a319b-613b-4eea-b074-506589384b55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:35:10.178464 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.178134 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "820a319b-613b-4eea-b074-506589384b55" (UID: "820a319b-613b-4eea-b074-506589384b55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:35:10.178653 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.178568 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "820a319b-613b-4eea-b074-506589384b55" (UID: "820a319b-613b-4eea-b074-506589384b55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:35:10.179902 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.179880 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820a319b-613b-4eea-b074-506589384b55-kube-api-access-82lcn" (OuterVolumeSpecName: "kube-api-access-82lcn") pod "820a319b-613b-4eea-b074-506589384b55" (UID: "820a319b-613b-4eea-b074-506589384b55"). InnerVolumeSpecName "kube-api-access-82lcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:35:10.179972 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.179942 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/820a319b-613b-4eea-b074-506589384b55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "820a319b-613b-4eea-b074-506589384b55" (UID: "820a319b-613b-4eea-b074-506589384b55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:35:10.278794 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.278772 2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:35:10.278871 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.278795 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/820a319b-613b-4eea-b074-506589384b55-kserve-provision-location\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:35:10.278871 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.278805 2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/820a319b-613b-4eea-b074-506589384b55-cabundle-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:35:10.278871 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.278814 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82lcn\" (UniqueName: \"kubernetes.io/projected/820a319b-613b-4eea-b074-506589384b55-kube-api-access-82lcn\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:35:10.278871 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:10.278823 2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/820a319b-613b-4eea-b074-506589384b55-proxy-tls\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:35:11.159531 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.159443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" event={"ID":"820a319b-613b-4eea-b074-506589384b55","Type":"ContainerDied","Data":"1e401dfdd98a0943880f019201a5a9df5637fc6d9f2a8abd04cebf6e5b5b7fac"} Jul 28 10:35:11.159531 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.159465 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk" Jul 28 10:35:11.159531 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.159531 2569 scope.go:117] "RemoveContainer" containerID="1cf9cd91000e298eb7b5a2683aecec36219e9249fa3e3ab81498c3ea3ac5b87c" Jul 28 10:35:11.161609 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.161582 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" event={"ID":"ded52cbb-42f4-4c5d-98a5-278c2e31547f","Type":"ContainerStarted","Data":"b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9"} Jul 28 10:35:11.161719 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.161675 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" event={"ID":"ded52cbb-42f4-4c5d-98a5-278c2e31547f","Type":"ContainerStarted","Data":"97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e"} Jul 28 10:35:11.161902 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.161886 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:11.169371 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.169357 2569 scope.go:117] "RemoveContainer" containerID="eda037efa84acbdcd3910cd85a88252059f58e9a8c8feaa96c6cbcc5e5f8447d" Jul 28 10:35:11.176717 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.176701 2569 scope.go:117] "RemoveContainer" containerID="90fa7b17a106f6748387bebc9e38e2b6f6d7a2879a3016a3d196b3cd78b07396" Jul 28 10:35:11.187324 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.187289 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podStartSLOduration=2.357426262 podStartE2EDuration="6.187277372s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.82985111s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:11.184820628 +0000 UTC m=+1655.627399604" watchObservedRunningTime="2026-07-28 10:35:11.187277372 +0000 UTC m=+1655.629856346" Jul 28 10:35:11.198446 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.198426 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk"] Jul 28 10:35:11.202581 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:11.202556 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7799dfb7df-lqqtk"] Jul 28 10:35:12.079097 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:12.079055 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="820a319b-613b-4eea-b074-506589384b55" path="/var/lib/kubelet/pods/820a319b-613b-4eea-b074-506589384b55/volumes" Jul 28 10:35:12.166369 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:12.166336 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:12.167435 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:12.167408 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:35:13.170910 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:13.170872 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:35:18.175722 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:18.175692 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:35:18.176215 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:18.176188 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:35:28.176911 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:28.176869 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:35:38.176975 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:38.176936 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:35:48.176863 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:48.176816 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:35:58.176259 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:35:58.176173 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:36:08.176193 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:08.176150 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:36:18.176606 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:18.176565 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:36:28.176799 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:28.176764 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:36:35.610526 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:35.610489 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q"] Jul 28 10:36:35.611014 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:35.610827 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" containerID="cri-o://97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e" gracePeriod=30 Jul 28 10:36:35.611014 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:35.610874 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kube-rbac-proxy" containerID="cri-o://b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9" gracePeriod=30 Jul 28 10:36:36.531321 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:36.531287 2569 generic.go:356] "Generic (PLEG): container finished" podID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerID="b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9" exitCode=2 Jul 28 10:36:36.531485 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:36.531357 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" event={"ID":"ded52cbb-42f4-4c5d-98a5-278c2e31547f","Type":"ContainerDied","Data":"b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9"} Jul 28 10:36:38.171079 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:38.171037 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused" Jul 28 10:36:38.176314 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:38.176289 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused" Jul 28 10:36:40.260011 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.259988 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:36:40.323446 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.323380 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-rbac-proxy-sar-config\") pod \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " Jul 28 10:36:40.323446 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.323415 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded52cbb-42f4-4c5d-98a5-278c2e31547f-proxy-tls\") pod \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " Jul 28 10:36:40.323446 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.323445 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kserve-provision-location\") pod \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " Jul 28 10:36:40.324005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.323461 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-cabundle-cert\") pod \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " Jul 28 10:36:40.324005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.323532 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gl5v\" (UniqueName: \"kubernetes.io/projected/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-api-access-4gl5v\") pod \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\" (UID: \"ded52cbb-42f4-4c5d-98a5-278c2e31547f\") " Jul 28 10:36:40.324005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.323767 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ded52cbb-42f4-4c5d-98a5-278c2e31547f" (UID: "ded52cbb-42f4-4c5d-98a5-278c2e31547f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 10:36:40.324005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.323803 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ded52cbb-42f4-4c5d-98a5-278c2e31547f" (UID: "ded52cbb-42f4-4c5d-98a5-278c2e31547f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:36:40.324005 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.323839 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ded52cbb-42f4-4c5d-98a5-278c2e31547f" (UID: "ded52cbb-42f4-4c5d-98a5-278c2e31547f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 10:36:40.325493 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.325462 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-api-access-4gl5v" (OuterVolumeSpecName: "kube-api-access-4gl5v") pod "ded52cbb-42f4-4c5d-98a5-278c2e31547f" (UID: "ded52cbb-42f4-4c5d-98a5-278c2e31547f"). InnerVolumeSpecName "kube-api-access-4gl5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 10:36:40.325493 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.325486 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ded52cbb-42f4-4c5d-98a5-278c2e31547f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ded52cbb-42f4-4c5d-98a5-278c2e31547f" (UID: "ded52cbb-42f4-4c5d-98a5-278c2e31547f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 10:36:40.424787 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.424755 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gl5v\" (UniqueName: \"kubernetes.io/projected/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-api-access-4gl5v\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:36:40.424787 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.424780 2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:36:40.424787 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.424790 2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded52cbb-42f4-4c5d-98a5-278c2e31547f-proxy-tls\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:36:40.425000 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.424799 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded52cbb-42f4-4c5d-98a5-278c2e31547f-kserve-provision-location\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:36:40.425000 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.424808 2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded52cbb-42f4-4c5d-98a5-278c2e31547f-cabundle-cert\") on node \"ip-10-0-135-126.ec2.internal\" DevicePath \"\"" Jul 28 10:36:40.551228 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.551195 2569 generic.go:356] "Generic (PLEG): container finished" podID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerID="97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e" exitCode=0 Jul 28 10:36:40.551388 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.551245 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" Jul 28 10:36:40.551388 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.551271 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" event={"ID":"ded52cbb-42f4-4c5d-98a5-278c2e31547f","Type":"ContainerDied","Data":"97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e"} Jul 28 10:36:40.551388 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.551331 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q" event={"ID":"ded52cbb-42f4-4c5d-98a5-278c2e31547f","Type":"ContainerDied","Data":"d0d79ed7bfcda8659a38fb5117f1ae3521e4636ff88447ba3fe59f0b4821b2c8"} Jul 28 10:36:40.551388 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.551363 2569 scope.go:117] "RemoveContainer" containerID="b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9" Jul 28 10:36:40.560079 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.560059 2569 scope.go:117] "RemoveContainer" containerID="97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e" Jul 28 10:36:40.567613 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.567592 2569 scope.go:117] "RemoveContainer" containerID="bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713" Jul 28 10:36:40.575117 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.575031 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q"] Jul 28 10:36:40.575306 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.575209 2569 scope.go:117] "RemoveContainer" containerID="b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9" Jul 28 10:36:40.575533 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:36:40.575510 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9\": container with ID starting with b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9 not found: ID does not exist" containerID="b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9" Jul 28 10:36:40.575662 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.575542 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9"} err="failed to get container status \"b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9\": rpc error: code = NotFound desc = could not find container \"b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9\": container with ID starting with b23dfdc47d1f1d61663597c4e9b1f69412a01410893686f35d4d5ae735f6bec9 not found: ID does not exist" Jul 28 10:36:40.575662 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.575565 2569 scope.go:117] "RemoveContainer" containerID="97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e" Jul 28 10:36:40.575876 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:36:40.575852 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e\": container with ID starting with 97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e not found: ID does not exist" containerID="97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e" Jul 28 10:36:40.575957 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.575879 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e"} err="failed to get container status \"97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e\": rpc error: code = NotFound desc = could not find container \"97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e\": container with ID starting with 97d967d9cab39a4a5747364d3184f0579e13b6ca7ed784986bb5130e7cde832e not found: ID does not exist" Jul 28 10:36:40.575957 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.575894 2569 scope.go:117] "RemoveContainer" containerID="bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713" Jul 28 10:36:40.576107 ip-10-0-135-126 kubenswrapper[2569]: E0728 10:36:40.576092 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713\": container with ID starting with bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713 not found: ID does not exist" containerID="bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713" Jul 28 10:36:40.576149 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.576111 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713"} err="failed to get container status \"bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713\": rpc error: code = NotFound desc = could not find container \"bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713\": container with ID starting with bb9dd77e2cf1391c491245db3b2d61bff06abfab304e909b77c94c969d50a713 not found: ID does not exist" Jul 28 10:36:40.579289 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:40.579267 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-bb95f9bbd-dzj7q"] Jul 28 10:36:42.079311 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:36:42.079273 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" path="/var/lib/kubelet/pods/ded52cbb-42f4-4c5d-98a5-278c2e31547f/volumes" Jul 28 10:37:36.052295 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:37:36.052266 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:37:36.053210 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:37:36.053188 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:42:36.078189 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:42:36.078142 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:42:36.080009 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:42:36.079987 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:44:18.983061 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:18.983034 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q6r8g_cbac8868-5bb5-4857-895f-455556932955/global-pull-secret-syncer/0.log" Jul 28 10:44:19.177830 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:19.177800 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r4n9j_e564183a-fced-4212-9fcf-6c1ffbf65cfa/konnectivity-agent/0.log" Jul 28 10:44:19.277275 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:19.277248 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-126.ec2.internal_05b32d16ed33041d1fceff9e2d07ef80/haproxy/0.log" Jul 28 10:44:23.718752 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:23.718723 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-s6m87_fcdc3068-bf6d-437d-9b29-1210b9eb6bd6/cluster-monitoring-operator/0.log" Jul 28 10:44:24.037889 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.037861 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xwvhp_e94544ef-0ac0-4cb8-a3ca-ba433190f28a/node-exporter/0.log" Jul 28 10:44:24.061220 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.061196 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xwvhp_e94544ef-0ac0-4cb8-a3ca-ba433190f28a/kube-rbac-proxy/0.log" Jul 28 10:44:24.086671 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.086649 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xwvhp_e94544ef-0ac0-4cb8-a3ca-ba433190f28a/init-textfile/0.log" Jul 28 10:44:24.372095 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.372017 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-2cvht_c240e311-cb8d-4abf-b063-e0dba541ebcc/prometheus-operator/0.log" Jul 28 10:44:24.396936 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.396913 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-2cvht_c240e311-cb8d-4abf-b063-e0dba541ebcc/kube-rbac-proxy/0.log" Jul 28 10:44:24.423239 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.423213 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-c5p5c_0d88a30e-5847-48db-a733-fc0e437d7980/prometheus-operator-admission-webhook/0.log" Jul 28 10:44:24.555907 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.555883 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-758cc57597-ktmkw_87951c92-a735-4cd3-b0ec-b486b04135c8/thanos-query/0.log" Jul 28 10:44:24.584525 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.584505 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-758cc57597-ktmkw_87951c92-a735-4cd3-b0ec-b486b04135c8/kube-rbac-proxy-web/0.log" Jul 28 10:44:24.608330 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.608312 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-758cc57597-ktmkw_87951c92-a735-4cd3-b0ec-b486b04135c8/kube-rbac-proxy/0.log" Jul 28 10:44:24.640354 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.640303 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-758cc57597-ktmkw_87951c92-a735-4cd3-b0ec-b486b04135c8/prom-label-proxy/0.log" Jul 28 10:44:24.680087 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.680069 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-758cc57597-ktmkw_87951c92-a735-4cd3-b0ec-b486b04135c8/kube-rbac-proxy-rules/0.log" Jul 28 10:44:24.713202 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:24.713176 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-758cc57597-ktmkw_87951c92-a735-4cd3-b0ec-b486b04135c8/kube-rbac-proxy-metrics/0.log" Jul 28 10:44:26.372805 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:26.372773 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/2.log" Jul 28 10:44:26.377072 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:26.377048 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-wkm97_fafae72a-446d-4852-9799-ed64ff1569fd/console-operator/3.log" Jul 28 10:44:26.770902 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:26.770874 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65fc554bd4-dcwpw_6e659632-57f5-4774-a888-145008a48b20/console/0.log" Jul 28 10:44:26.809479 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:26.804724 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-2wp29_9669856d-bc18-4b5c-8e4f-1398d95653ab/download-server/0.log" Jul 28 10:44:27.278545 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278511 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w"] Jul 28 10:44:27.278892 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278878 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kube-rbac-proxy" Jul 28 10:44:27.278938 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278894 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kube-rbac-proxy" Jul 28 10:44:27.278938 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278905 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="storage-initializer" Jul 28 10:44:27.278938 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278911 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="storage-initializer" Jul 28 10:44:27.278938 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278928 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="storage-initializer" Jul 28 10:44:27.278938 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278935 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="storage-initializer" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278941 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278947 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278953 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278958 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278965 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kube-rbac-proxy" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.278970 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kube-rbac-proxy" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.279022 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kube-rbac-proxy" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.279029 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kserve-container" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.279034 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="820a319b-613b-4eea-b074-506589384b55" containerName="kube-rbac-proxy" Jul 28 10:44:27.279320 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.279041 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded52cbb-42f4-4c5d-98a5-278c2e31547f" containerName="kserve-container" Jul 28 10:44:27.281751 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.281734 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.283197 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.283171 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-npsmw_056218d8-cf3f-4415-99c2-cac7b557f301/volume-data-source-validator/0.log" Jul 28 10:44:27.284728 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.284706 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pp4wj\"/\"openshift-service-ca.crt\"" Jul 28 10:44:27.284728 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.284726 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pp4wj\"/\"kube-root-ca.crt\"" Jul 28 10:44:27.285789 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.285770 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pp4wj\"/\"default-dockercfg-w6bp4\"" Jul 28 10:44:27.290849 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.290829 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w"] Jul 28 10:44:27.392576 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.392539 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-proc\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.392576 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.392576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-lib-modules\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.393046 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.392595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-sys\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.393046 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.392689 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-podres\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.393046 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.392720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g2nj\" (UniqueName: \"kubernetes.io/projected/75bfeab4-f2b3-493d-b797-7f1b51d90596-kube-api-access-5g2nj\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.493391 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.493350 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-proc\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.493391 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.493391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-lib-modules\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.493610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.493412 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-sys\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.493610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.493436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-podres\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.493610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.493462 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g2nj\" (UniqueName: \"kubernetes.io/projected/75bfeab4-f2b3-493d-b797-7f1b51d90596-kube-api-access-5g2nj\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.493610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.493487 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-proc\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.493610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.493523 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-sys\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.493610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.493559 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-podres\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.493610 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.493559 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75bfeab4-f2b3-493d-b797-7f1b51d90596-lib-modules\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.501228 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.501204 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g2nj\" (UniqueName: \"kubernetes.io/projected/75bfeab4-f2b3-493d-b797-7f1b51d90596-kube-api-access-5g2nj\") pod \"perf-node-gather-daemonset-z997w\" (UID: \"75bfeab4-f2b3-493d-b797-7f1b51d90596\") " pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.594452 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.594372 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:27.721930 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:27.721902 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w"] Jul 28 10:44:27.724391 ip-10-0-135-126 kubenswrapper[2569]: W0728 10:44:27.724363 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod75bfeab4_f2b3_493d_b797_7f1b51d90596.slice/crio-54cc9636ac543588a705865cd89da545c7f36db2031b867789fa0c064400ae4a WatchSource:0}: Error finding container 54cc9636ac543588a705865cd89da545c7f36db2031b867789fa0c064400ae4a: Status 404 returned error can't find the container with id 54cc9636ac543588a705865cd89da545c7f36db2031b867789fa0c064400ae4a Jul 28 10:44:28.261174 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:28.261148 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gdk8g_5e8ba7d9-892e-4f45-846e-9d56500b25de/dns/0.log" Jul 28 10:44:28.284817 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:28.284793 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gdk8g_5e8ba7d9-892e-4f45-846e-9d56500b25de/kube-rbac-proxy/0.log" Jul 28 10:44:28.450104 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:28.450072 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" event={"ID":"75bfeab4-f2b3-493d-b797-7f1b51d90596","Type":"ContainerStarted","Data":"d391a8d12dca81b9714585ab84009b9dc161d422ffc777de8da32943a4362542"} Jul 28 10:44:28.450492 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:28.450123 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" event={"ID":"75bfeab4-f2b3-493d-b797-7f1b51d90596","Type":"ContainerStarted","Data":"54cc9636ac543588a705865cd89da545c7f36db2031b867789fa0c064400ae4a"} Jul 28 10:44:28.450492 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:28.450201 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:28.454948 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:28.454926 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tp929_022e6ace-b742-4eb3-a13a-97d1ec1f10e3/dns-node-resolver/0.log" Jul 28 10:44:28.466017 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:28.465980 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" podStartSLOduration=1.465966788 podStartE2EDuration="1.465966788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:44:28.464919613 +0000 UTC m=+2212.907498587" watchObservedRunningTime="2026-07-28 10:44:28.465966788 +0000 UTC m=+2212.908545782" Jul 28 10:44:28.918657 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:28.918614 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mj48w_1acd164c-032f-40eb-bef6-b07042848a90/node-ca/0.log" Jul 28 10:44:29.699272 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:29.699241 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b5bf78f8f-wkjq5_0d7c4780-07fc-4890-b1d9-ec4b1a2cd1ac/router/0.log" Jul 28 10:44:30.000035 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:30.000004 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b59sc_c1bf9f83-7660-43f9-8827-79237d176fcb/serve-healthcheck-canary/0.log" Jul 28 10:44:30.368528 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:30.368453 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-jd9hz_6ba36116-0fd9-4d91-b070-c52a5378dc54/insights-operator/0.log" Jul 28 10:44:30.369604 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:30.369585 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-jd9hz_6ba36116-0fd9-4d91-b070-c52a5378dc54/insights-operator/1.log" Jul 28 10:44:30.517065 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:30.517037 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q5dpw_de963509-197c-4ed1-af8b-afecc19b465e/kube-rbac-proxy/0.log" Jul 28 10:44:30.537013 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:30.536990 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q5dpw_de963509-197c-4ed1-af8b-afecc19b465e/exporter/0.log" Jul 28 10:44:30.557076 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:30.557053 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q5dpw_de963509-197c-4ed1-af8b-afecc19b465e/extractor/0.log" Jul 28 10:44:32.776798 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:32.776762 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6968749498-5zm2t_2dd44e98-7fd3-48f0-acd0-d409409bcad7/manager/0.log" Jul 28 10:44:32.938728 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:32.938694 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-djbtd_9f4c9c7b-9f2b-4719-9bb9-ad57f9d195f9/manager/0.log" Jul 28 10:44:33.014386 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:33.014359 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-jqhcq_22c7897f-0234-4331-99f3-0e931bffcc4b/seaweedfs/0.log" Jul 28 10:44:34.463086 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:34.463061 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pp4wj/perf-node-gather-daemonset-z997w" Jul 28 10:44:37.463806 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:37.463716 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr_25239465-8177-4ac4-9cb3-a293eb83589e/kube-storage-version-migrator-operator/1.log" Jul 28 10:44:37.464607 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:37.464590 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-q5vdr_25239465-8177-4ac4-9cb3-a293eb83589e/kube-storage-version-migrator-operator/0.log" Jul 28 10:44:38.500289 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:38.500262 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75jhn_dea71310-9b24-40e8-8b58-8780a189b3df/kube-multus-additional-cni-plugins/0.log" Jul 28 10:44:38.526892 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:38.526866 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75jhn_dea71310-9b24-40e8-8b58-8780a189b3df/egress-router-binary-copy/0.log" Jul 28 10:44:38.549852 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:38.549823 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75jhn_dea71310-9b24-40e8-8b58-8780a189b3df/cni-plugins/0.log" Jul 28 10:44:38.572292 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:38.572272 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75jhn_dea71310-9b24-40e8-8b58-8780a189b3df/bond-cni-plugin/0.log" Jul 28 10:44:38.597285 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:38.597264 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75jhn_dea71310-9b24-40e8-8b58-8780a189b3df/routeoverride-cni/0.log" Jul 28 10:44:38.638450 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:38.638427 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75jhn_dea71310-9b24-40e8-8b58-8780a189b3df/whereabouts-cni-bincopy/0.log" Jul 28 10:44:38.684982 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:38.684958 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-75jhn_dea71310-9b24-40e8-8b58-8780a189b3df/whereabouts-cni/0.log" Jul 28 10:44:39.016413 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:39.016388 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z899d_7062085c-ecbf-476b-b845-e594877f19d9/kube-multus/0.log" Jul 28 10:44:39.131294 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:39.131266 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-trrp4_06b28783-48ec-4b28-94de-385cd15baaa8/network-metrics-daemon/0.log" Jul 28 10:44:39.152596 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:39.152531 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-trrp4_06b28783-48ec-4b28-94de-385cd15baaa8/kube-rbac-proxy/0.log" Jul 28 10:44:40.220130 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:40.220099 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wkd29_12f63461-f09b-4642-b556-023c9f371845/ovn-controller/0.log" Jul 28 10:44:40.237406 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:40.237383 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wkd29_12f63461-f09b-4642-b556-023c9f371845/ovn-acl-logging/0.log" Jul 28 10:44:40.259037 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:40.259015 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wkd29_12f63461-f09b-4642-b556-023c9f371845/kube-rbac-proxy-node/0.log" Jul 28 10:44:40.277911 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:40.277887 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wkd29_12f63461-f09b-4642-b556-023c9f371845/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 10:44:40.296713 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:40.296691 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wkd29_12f63461-f09b-4642-b556-023c9f371845/northd/0.log" Jul 28 10:44:40.315324 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:40.315307 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wkd29_12f63461-f09b-4642-b556-023c9f371845/nbdb/0.log" Jul 28 10:44:40.336631 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:40.336605 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wkd29_12f63461-f09b-4642-b556-023c9f371845/sbdb/0.log" Jul 28 10:44:40.376850 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:40.376830 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wkd29_12f63461-f09b-4642-b556-023c9f371845/ovnkube-controller/0.log" Jul 28 10:44:41.378850 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:41.378817 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fr92k_d6b7677d-6c6c-4be6-893c-936e9ca050be/network-check-target-container/0.log" Jul 28 10:44:42.233992 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:42.233967 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qdg7w_333ddf02-969b-40cf-8d27-bb408a5a3873/iptables-alerter/0.log" Jul 28 10:44:42.822260 ip-10-0-135-126 kubenswrapper[2569]: I0728 10:44:42.822233 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-b2j8p_19bdd7fe-0327-4db4-83ad-38b5b334cbdc/tuned/0.log"