Jul 07 14:54:29.198906 ip-10-0-130-13 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 14:54:29.198919 ip-10-0-130-13 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 14:54:29.198929 ip-10-0-130-13 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 14:54:29.199248 ip-10-0-130-13 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 14:54:39.447943 ip-10-0-130-13 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 14:54:39.447964 ip-10-0-130-13 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a41b79389af6451e91e21d3f808a637a -- Jul 07 14:56:31.264552 ip-10-0-130-13 systemd[1]: Starting Kubernetes Kubelet... Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.674317 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679302 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679315 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679318 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679321 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679324 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679327 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679331 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:56:31.739015 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679333 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679336 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679341 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679345 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679348 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679350 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679353 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679359 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679362 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679365 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679368 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679370 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679373 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679381 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679384 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679387 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679389 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679392 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679394 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:56:31.740024 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679397 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679400 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679403 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679405 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679408 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679411 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679413 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679416 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679419 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679421 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679424 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679426 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679436 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679438 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679441 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679443 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679446 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679448 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679452 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:56:31.740830 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679455 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679458 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679461 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679464 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679467 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679469 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679472 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679474 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679477 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679480 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679482 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679484 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679487 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679489 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679492 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679495 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679498 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679502 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679506 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679509 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:56:31.741519 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679512 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679514 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679518 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679520 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679523 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679526 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679534 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679536 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679539 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679542 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679544 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679546 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679549 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679551 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679554 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679556 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679559 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679561 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679564 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:56:31.742084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679566 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679568 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679571 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679573 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679576 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679578 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679580 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679583 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679585 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679588 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679590 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679593 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679596 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679598 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.679601 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680356 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680363 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680366 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680368 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680371 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:56:31.742611 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680374 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680377 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680379 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680382 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680385 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680388 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680391 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680393 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680396 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680398 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680400 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680404 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680406 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680409 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680411 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680414 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680416 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680419 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680421 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:56:31.743154 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680424 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680426 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680429 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680431 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680434 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680437 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680439 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680442 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680445 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680447 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680450 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680452 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680454 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680457 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680461 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680464 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680466 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680469 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680472 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680475 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:56:31.743718 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680477 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680479 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680482 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680484 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680487 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680489 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680492 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680494 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680497 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680500 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680504 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680509 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680513 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680516 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680520 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680523 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680525 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680528 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680531 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:56:31.744319 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680533 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680536 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680538 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680541 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680543 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680545 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680548 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680552 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680555 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680557 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680560 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680563 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680565 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680568 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680570 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680573 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680575 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680578 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680580 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680582 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:56:31.744947 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680585 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680588 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680591 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680593 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680596 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680598 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680600 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680603 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680605 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680608 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680610 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680613 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680615 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680617 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680619 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680622 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680976 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680982 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680985 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680988 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:56:31.745696 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680991 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680994 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680996 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.680999 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681001 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681005 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681008 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681010 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681012 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681016 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681019 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681022 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681025 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681027 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681029 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681032 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681034 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681037 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681039 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681042 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:56:31.746215 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681044 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681047 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681049 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681053 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681056 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681059 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681061 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681064 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681067 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681070 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681073 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681075 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681078 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681081 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681084 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681086 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681089 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681092 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681095 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:56:31.746893 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681097 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681100 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681102 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681107 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681110 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681113 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681115 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681118 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681120 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681122 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681125 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681127 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681130 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681132 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681135 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681137 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681140 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681142 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681145 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:56:31.747528 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681147 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681149 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681152 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681155 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681158 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681160 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681163 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681168 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681171 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681173 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681175 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681179 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681181 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681184 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681186 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681190 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681192 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681195 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681197 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681199 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:56:31.748642 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681202 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:56:31.748416 ip-10-0-130-13 systemd[1]: Started Kubernetes Kubelet. Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681204 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681207 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681209 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681212 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681214 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681217 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681219 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681222 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681224 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681227 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681229 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681232 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681234 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681236 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681240 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681243 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681322 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681341 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681347 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681352 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 14:56:31.749672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681357 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681360 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681365 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681369 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681372 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681375 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681378 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681381 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681384 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681387 2566 flags.go:64] FLAG: --cgroup-root="" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681390 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681392 2566 flags.go:64] FLAG: --client-ca-file="" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681395 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681398 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681402 2566 flags.go:64] FLAG: --cluster-domain="" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681404 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681407 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681410 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681413 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681416 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681419 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681422 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681425 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681428 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 07 14:56:31.750941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681430 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681433 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681437 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681439 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681443 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681446 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681449 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681452 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681454 2566 flags.go:64] FLAG: --enable-server="true" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681457 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681463 2566 flags.go:64] FLAG: --event-burst="100" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681466 2566 flags.go:64] FLAG: --event-qps="50" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681469 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681472 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681475 2566 flags.go:64] FLAG: --eviction-hard="" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681478 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681481 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681483 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681486 2566 flags.go:64] FLAG: --eviction-soft="" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681489 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681492 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681495 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681497 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681500 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681502 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 14:56:31.751967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681505 2566 flags.go:64] FLAG: --feature-gates="" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681509 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681512 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681515 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681517 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681520 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681523 2566 flags.go:64] FLAG: --help="false" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681526 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-130-13.ec2.internal" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681528 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681532 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681535 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681538 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681541 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681544 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681546 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681549 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681552 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681556 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681559 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681562 2566 flags.go:64] FLAG: --kube-reserved="" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681565 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681568 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681571 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681573 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 14:56:31.753646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681576 2566 flags.go:64] FLAG: --lock-file="" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681578 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681581 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681584 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681589 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681592 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681594 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681597 2566 flags.go:64] FLAG: --logging-format="text" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681600 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681603 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681606 2566 flags.go:64] FLAG: --manifest-url="" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681608 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681616 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681619 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681623 2566 flags.go:64] FLAG: --max-pods="110" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681626 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681629 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681634 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681637 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681640 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681643 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681646 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681652 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681655 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681658 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 14:56:31.755168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681662 2566 flags.go:64] FLAG: --pod-cidr="" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681666 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681671 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681673 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681676 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681679 2566 flags.go:64] FLAG: --port="10250" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681682 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681684 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08640c77aa8e92ac5" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681687 2566 flags.go:64] FLAG: --qos-reserved="" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681691 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681693 2566 flags.go:64] FLAG: --register-node="true" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681696 2566 flags.go:64] FLAG: --register-with-taints="" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681699 2566 flags.go:64] FLAG: --registry-burst="10" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681702 2566 flags.go:64] FLAG: --registry-qps="5" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681704 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681707 2566 flags.go:64] FLAG: --reserved-memory="" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681710 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681713 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681716 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681719 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681721 2566 flags.go:64] FLAG: --runonce="false" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681724 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681726 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681729 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681733 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 14:56:31.904147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681735 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681738 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681741 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681744 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681747 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681749 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681752 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681755 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681759 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681762 2566 flags.go:64] FLAG: --system-cgroups="" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681765 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681770 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681773 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681775 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681779 2566 flags.go:64] FLAG: --tls-min-version="" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681782 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681784 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681787 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681789 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681792 2566 flags.go:64] FLAG: --v="2" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681796 2566 flags.go:64] FLAG: --version="false" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681799 2566 flags.go:64] FLAG: --vmodule="" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681803 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.681806 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681912 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:56:31.923084 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681916 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681919 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681922 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681925 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681927 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681930 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681934 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681938 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681941 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681944 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681946 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681949 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681952 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681954 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681957 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681961 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681963 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681968 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681971 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681973 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:56:31.924065 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681976 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681978 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681983 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681985 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681988 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681990 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681994 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.681997 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682000 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682003 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682006 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682008 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682011 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682013 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682016 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682018 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682021 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682023 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682027 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:56:31.924879 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682030 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682032 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682035 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682037 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682040 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682042 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682045 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682047 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682053 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682057 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682060 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682063 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682066 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682069 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682071 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682074 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682077 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682079 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682081 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:56:31.925534 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682084 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682086 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682089 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682091 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682094 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682097 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682099 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682102 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682104 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682107 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682109 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682111 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682115 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682118 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682121 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682123 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682126 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682128 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682131 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682133 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:56:31.926094 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682136 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682140 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682142 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682144 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682147 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682150 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682152 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682155 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682157 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682159 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682162 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682164 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682166 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682169 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682171 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682174 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682176 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682179 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682181 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:56:31.926747 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.682183 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.682189 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.690657 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.690682 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.690700 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.690705 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690763 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690768 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690771 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690774 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690777 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690780 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690783 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690786 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690788 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:56:31.927325 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690791 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690793 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690796 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690799 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690801 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690804 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690807 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690810 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690812 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690815 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690818 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690820 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690823 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690825 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690828 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690831 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690834 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690836 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690840 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690842 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:56:31.927802 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690845 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690847 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690850 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690853 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690855 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690858 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690860 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690863 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690865 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690868 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690871 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690873 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690876 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690879 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690881 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690883 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690888 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690891 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690894 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690896 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:56:31.928347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690899 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690904 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690906 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690909 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690911 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690914 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690917 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690920 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690922 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690925 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690928 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690930 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690933 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690936 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690939 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690943 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690946 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690949 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690951 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:56:31.928908 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690954 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690956 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690959 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690961 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690964 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690966 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690969 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690971 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690974 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690976 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690979 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690982 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690984 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690988 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690994 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690997 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.690999 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691002 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691005 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:56:31.929435 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691008 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691010 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691015 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691018 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691020 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691022 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691025 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691027 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691030 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691032 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691035 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691038 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.691043 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691163 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691168 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:56:31.929974 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691171 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691175 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691179 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691181 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691184 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691186 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691189 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691192 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691194 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691197 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691200 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691203 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691205 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691209 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691212 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691216 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691219 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691222 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691225 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:56:31.930416 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691229 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691232 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691235 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691238 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691241 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691244 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691247 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691249 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691252 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691254 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691257 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691259 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691262 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691278 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691283 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691285 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691288 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691290 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691293 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:56:31.930941 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691295 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691298 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691300 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691303 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691305 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691309 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691311 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691315 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691318 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691320 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691323 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691325 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691327 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691331 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691334 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691336 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691339 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691341 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691344 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691346 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:56:31.931474 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691349 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691351 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691354 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691357 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691359 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691361 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691364 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691366 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691368 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691371 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691373 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691376 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691378 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691380 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691383 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691385 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691388 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691390 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691393 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691397 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:56:31.932045 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691399 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691401 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691404 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691406 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691409 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691416 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691418 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691420 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691423 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691426 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691428 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691431 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691433 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691436 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691439 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691441 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691444 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691446 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:56:31.932639 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:31.691448 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.691454 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.691593 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.696088 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.697475 2566 server.go:1019] "Starting client certificate rotation" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.697573 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.697634 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.701171 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.702256 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.713865 2566 log.go:25] "Validated CRI v1 runtime API" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.716082 2566 log.go:25] "Validated CRI v1 image API" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.716785 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 14:56:32.100664 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.720978 2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 89d61456-d361-4c24-9a35-8c7772652b67:/dev/nvme0n1p4 f2a98e3a-038c-42e1-be66-9beb9119a2a2:/dev/nvme0n1p3] Jul 07 14:56:32.101145 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.720991 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 14:56:32.101341 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.726853 2566 manager.go:216] Machine: {Timestamp:2026-07-07 14:56:31.724693478 +0000 UTC m=+0.337528041 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3197820 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22248e7b35efdcdb9126f6b50053c9 SystemUUID:ec22248e-7b35-efdc-db91-26f6b50053c9 BootID:a41b7938-9af6-451e-91e2-1d3f808a637a Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:b8:98:c0:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:b8:98:c0:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:56:db:1a:ad:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 14:56:32.101341 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.726955 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 14:56:32.101341 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.727030 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 14:56:32.101341 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.727649 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:56:32.101341 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.729977 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 14:56:32.101341 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.729998 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-13.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 14:56:32.101341 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.731327 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.731337 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.731345 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.731974 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.733301 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.736804 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.736991 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.737021 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.737033 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.737057 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.737075 2566 kubelet.go:398] "Adding apiserver pod source" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.737097 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.737496 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.737521 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.740380 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.742351 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.742379 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743476 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743489 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743494 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743499 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743506 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743512 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743520 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743525 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743532 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743537 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743544 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.743549 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.744412 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.744419 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.747603 2566 server.go:1268] "Started kubelet" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.747685 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.747849 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.747910 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.749920 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.751695 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.753467 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.753519 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-13.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.753596 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 14:56:32.102683 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.754459 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.756184 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.756195 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.756294 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.756831 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.756838 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.756855 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.756900 2566 factory.go:55] Registering systemd factory Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.756932 2566 factory.go:223] Registration of the systemd container factory successfully Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.756933 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.756943 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.757107 2566 factory.go:153] Registering CRI-O factory Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.757121 2566 factory.go:223] Registration of the crio container factory successfully Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.757163 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.757184 2566 factory.go:103] Registering Raw factory Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.757196 2566 manager.go:1196] Started watching for new ooms in manager Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.757631 2566 manager.go:349] Starting recovery of all containers Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.760476 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-13.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.760536 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.760618 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-13.ec2.internal.18c009c470b08dbd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-13.ec2.internal,UID:ip-10-0-130-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-13.ec2.internal,},FirstTimestamp:2026-07-07 14:56:31.747575229 +0000 UTC m=+0.360409792,LastTimestamp:2026-07-07 14:56:31.747575229 +0000 UTC m=+0.360409792,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-13.ec2.internal,}" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.765249 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.768303 2566 manager.go:354] Recovery completed Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.774525 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.779829 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.779871 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.779886 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.780493 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.780502 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.780518 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.780503 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.781353 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.781377 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:56:32.103794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.781388 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.782459 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-13.ec2.internal.18c009c4729d1968 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-13.ec2.internal,UID:ip-10-0-130-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-13.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-13.ec2.internal,},FirstTimestamp:2026-07-07 14:56:31.779854696 +0000 UTC m=+0.392689259,LastTimestamp:2026-07-07 14:56:31.779854696 +0000 UTC m=+0.392689259,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-13.ec2.internal,}" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.786025 2566 policy_none.go:49] "None policy: Start" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.786040 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.786054 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.787229 2566 policy_none.go:47] "Start" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.791128 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-13.ec2.internal.18c009c4729d74ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-13.ec2.internal,UID:ip-10-0-130-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-13.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-13.ec2.internal,},FirstTimestamp:2026-07-07 14:56:31.779878059 +0000 UTC m=+0.392712622,LastTimestamp:2026-07-07 14:56:31.779878059 +0000 UTC m=+0.392712622,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-13.ec2.internal,}" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.792059 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mgwpb" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.793685 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-13.ec2.internal.18c009c4729da71b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-13.ec2.internal,UID:ip-10-0-130-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-13.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-13.ec2.internal,},FirstTimestamp:2026-07-07 14:56:31.779890971 +0000 UTC m=+0.392725534,LastTimestamp:2026-07-07 14:56:31.779890971 +0000 UTC m=+0.392725534,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-13.ec2.internal,}" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.802107 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mgwpb" Jul 07 14:56:33.645314 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.802180 2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-13.ec2.internal.18c009c4729d1968\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-13.ec2.internal.18c009c4729d1968 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-13.ec2.internal,UID:ip-10-0-130-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-13.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-13.ec2.internal,},FirstTimestamp:2026-07-07 14:56:31.779854696 +0000 UTC m=+0.392689259,LastTimestamp:2026-07-07 14:56:31.781365356 +0000 UTC m=+0.394199917,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-13.ec2.internal,}" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.805128 2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-13.ec2.internal.18c009c4729d74ab\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-13.ec2.internal.18c009c4729d74ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-13.ec2.internal,UID:ip-10-0-130-13.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-13.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-13.ec2.internal,},FirstTimestamp:2026-07-07 14:56:31.779878059 +0000 UTC m=+0.392712622,LastTimestamp:2026-07-07 14:56:31.781382596 +0000 UTC m=+0.394217157,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-13.ec2.internal,}" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.831764 2566 manager.go:337] "Starting Device Plugin manager" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.831790 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.831800 2566 server.go:87] "Starting device plugin registration server" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.832047 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.832058 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.832151 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.832213 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.832222 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.833538 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.833576 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.900602 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.900640 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.900663 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.900694 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.902960 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.933021 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.933855 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.933883 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.933893 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.933917 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:31.942878 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.942897 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-13.ec2.internal\": node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:31.976147 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.001203 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal"] Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.001293 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.002229 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.002262 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.002289 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:56:33.646152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.003676 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.003786 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.003813 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.004400 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.004424 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.004432 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.004401 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.004495 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.004506 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.005971 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.005990 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.006651 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.006678 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.006692 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:32.042821 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-13.ec2.internal\" not found" node="ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:32.047163 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-13.ec2.internal\" not found" node="ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:32.076465 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.157885 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2a0f84d43a03da466d44795084f74c7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal\" (UID: \"d2a0f84d43a03da466d44795084f74c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.157908 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2a0f84d43a03da466d44795084f74c7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal\" (UID: \"d2a0f84d43a03da466d44795084f74c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:32.157925 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/952eb1cd57023c1b0bde0ddde055ed58-config\") pod \"kube-apiserver-proxy-ip-10-0-130-13.ec2.internal\" (UID: \"952eb1cd57023c1b0bde0ddde055ed58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.646992 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:32.176934 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:33.664308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2a0f84d43a03da466d44795084f74c7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal\" (UID: \"d2a0f84d43a03da466d44795084f74c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:33.664720 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2a0f84d43a03da466d44795084f74c7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal\" (UID: \"d2a0f84d43a03da466d44795084f74c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:33.664781 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/952eb1cd57023c1b0bde0ddde055ed58-config\") pod \"kube-apiserver-proxy-ip-10-0-130-13.ec2.internal\" (UID: \"952eb1cd57023c1b0bde0ddde055ed58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:33.665359 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/952eb1cd57023c1b0bde0ddde055ed58-config\") pod \"kube-apiserver-proxy-ip-10-0-130-13.ec2.internal\" (UID: \"952eb1cd57023c1b0bde0ddde055ed58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:33.665394 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2a0f84d43a03da466d44795084f74c7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal\" (UID: \"d2a0f84d43a03da466d44795084f74c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:33.665412 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2a0f84d43a03da466d44795084f74c7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal\" (UID: \"d2a0f84d43a03da466d44795084f74c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:33.682559 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:33.750724 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:33.754437 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal" Jul 07 14:56:33.789135 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:33.782782 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:33.923674 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:33.883639 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:34.084272 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:33.984452 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:34.085134 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:34.084923 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:34.232018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.102083 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 14:56:34.232018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.102220 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 14:56:34.232018 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:34.187127 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:34.232018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.210305 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:51:33 +0000 UTC" deadline="2028-02-28 21:52:20.388318528 +0000 UTC" Jul 07 14:56:34.232018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.210332 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14430h55m46.177989907s" Jul 07 14:56:34.232018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.222445 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:56:34.232018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.224275 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:56:34.232018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.224310 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:56:34.232018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.224321 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:56:34.384405 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:34.287963 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:34.384405 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.341672 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:56:34.537294 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:34.388801 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:34.537294 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:34.430186 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952eb1cd57023c1b0bde0ddde055ed58.slice/crio-5aba332f8897cb26ee70ee45a12ceeb730900c29a66cbef91da0d192e47f76ee WatchSource:0}: Error finding container 5aba332f8897cb26ee70ee45a12ceeb730900c29a66cbef91da0d192e47f76ee: Status 404 returned error can't find the container with id 5aba332f8897cb26ee70ee45a12ceeb730900c29a66cbef91da0d192e47f76ee Jul 07 14:56:34.537294 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:34.430508 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a0f84d43a03da466d44795084f74c7.slice/crio-99fd9fdd2923b246793e612d3c0b9e773588d8de187093b1c58fa382892f446c WatchSource:0}: Error finding container 99fd9fdd2923b246793e612d3c0b9e773588d8de187093b1c58fa382892f446c: Status 404 returned error can't find the container with id 99fd9fdd2923b246793e612d3c0b9e773588d8de187093b1c58fa382892f446c Jul 07 14:56:34.537294 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.434159 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:56:34.537294 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:34.489783 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-13.ec2.internal\" not found" Jul 07 14:56:34.712719 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.547734 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:56:34.712719 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.562065 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" Jul 07 14:56:34.712719 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.577333 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:56:34.712719 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.578685 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal" Jul 07 14:56:34.712719 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.588356 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:56:34.712719 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:34.649779 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:56:35.089774 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.076534 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.143248 2566 apiserver.go:52] "Watching apiserver" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.152259 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.152686 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jw459","openshift-network-diagnostics/network-check-target-mvn5b","openshift-network-operator/iptables-alerter-fbwxs","openshift-ovn-kubernetes/ovnkube-node-lw2hn","kube-system/global-pull-secret-syncer-kwswc","kube-system/konnectivity-agent-xckkk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg","openshift-cluster-node-tuning-operator/tuned-vg6bn","openshift-dns/node-resolver-7f74m","openshift-image-registry/node-ca-dlvgv","openshift-multus/multus-additional-cni-plugins-mn7mm","openshift-multus/multus-cfcxp","kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal"] Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.155248 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.155328 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.157391 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.157462 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.159588 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.161814 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.162674 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.163473 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w748q\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.163483 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.163532 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.163476 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.164076 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.164131 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.165186 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7mkvp\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.165323 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.166695 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.166907 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.167100 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.167321 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.167511 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 14:56:35.180984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.167834 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.168995 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.170747 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.171085 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xxpw4\"" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.171822 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.172120 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.172704 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.173198 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.173310 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sp2xw\"" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179362 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-systemd-units\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179391 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-cni-netd\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179395 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-kubelet\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179435 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-var-lib-openvswitch\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179460 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-run-ovn\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179484 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-run-ovn-kubernetes\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179507 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-cni-bin\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179529 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0678b753-d3d1-4fd9-950d-eec1f33fcc86-ovn-node-metrics-cert\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179568 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-slash\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.181959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179595 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-run-systemd\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179626 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-node-log\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179644 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-log-socket\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179667 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179687 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-etc-openvswitch\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmk4r\" (UniqueName: \"kubernetes.io/projected/0678b753-d3d1-4fd9-950d-eec1f33fcc86-kube-api-access-cmk4r\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179749 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0678b753-d3d1-4fd9-950d-eec1f33fcc86-ovnkube-script-lib\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179773 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d998f795-7247-4ec8-ab81-9e15fc38d814-host-slash\") pod \"iptables-alerter-fbwxs\" (UID: \"d998f795-7247-4ec8-ab81-9e15fc38d814\") " pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179816 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-run-openvswitch\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95kgh\" (UniqueName: \"kubernetes.io/projected/f115fbf4-80c0-4991-b663-61ce83d6e968-kube-api-access-95kgh\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179854 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-run-netns\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179873 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0678b753-d3d1-4fd9-950d-eec1f33fcc86-ovnkube-config\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.182687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179892 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:35.183520 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8w8p\" (UniqueName: \"kubernetes.io/projected/d998f795-7247-4ec8-ab81-9e15fc38d814-kube-api-access-n8w8p\") pod \"iptables-alerter-fbwxs\" (UID: \"d998f795-7247-4ec8-ab81-9e15fc38d814\") " pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.183520 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179928 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0678b753-d3d1-4fd9-950d-eec1f33fcc86-env-overrides\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.183520 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.179947 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d998f795-7247-4ec8-ab81-9e15fc38d814-iptables-alerter-script\") pod \"iptables-alerter-fbwxs\" (UID: \"d998f795-7247-4ec8-ab81-9e15fc38d814\") " pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.184633 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.184106 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.187820 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.187455 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.188237 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.188216 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 14:56:35.188855 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.188838 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jgx76\"" Jul 07 14:56:35.189017 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.188989 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:56:35.195514 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.191244 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.195514 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.191629 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9jnxw\"" Jul 07 14:56:35.195514 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.193457 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.195514 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.194581 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 14:56:35.196980 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.196128 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 14:56:35.196980 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.196327 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 14:56:35.196980 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.196593 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.197357 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.197018 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 14:56:35.197357 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.197155 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 14:56:35.197357 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.197158 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zctwf\"" Jul 07 14:56:35.200479 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.199519 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 14:56:35.200479 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.199739 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 14:56:35.200479 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.199757 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mlrxl\"" Jul 07 14:56:35.200479 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.199888 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 14:56:35.200479 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.200021 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 14:56:35.200479 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.200189 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 14:56:35.200479 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.200215 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 14:56:35.200479 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.200351 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zkj7z\"" Jul 07 14:56:35.203548 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.203267 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbk5z" Jul 07 14:56:35.211829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.211718 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbk5z" Jul 07 14:56:35.263110 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.263090 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 14:56:35.280446 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280386 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95kgh\" (UniqueName: \"kubernetes.io/projected/f115fbf4-80c0-4991-b663-61ce83d6e968-kube-api-access-95kgh\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:35.280446 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280433 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-run-netns\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.280579 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0678b753-d3d1-4fd9-950d-eec1f33fcc86-ovnkube-config\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.280579 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280485 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-sysctl-d\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.280579 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280526 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-registration-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.280579 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:35.280579 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280577 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8w8p\" (UniqueName: \"kubernetes.io/projected/d998f795-7247-4ec8-ab81-9e15fc38d814-kube-api-access-n8w8p\") pod \"iptables-alerter-fbwxs\" (UID: \"d998f795-7247-4ec8-ab81-9e15fc38d814\") " pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280604 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-var-lib-kubelet\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280627 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280651 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-run-k8s-cni-cncf-io\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280676 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d998f795-7247-4ec8-ab81-9e15fc38d814-iptables-alerter-script\") pod \"iptables-alerter-fbwxs\" (UID: \"d998f795-7247-4ec8-ab81-9e15fc38d814\") " pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280700 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-systemd-units\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280730 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-cni-netd\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280750 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-os-release\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280773 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-socket-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280796 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-kubelet\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.280819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-var-lib-openvswitch\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280844 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-run-ovn-kubernetes\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280868 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-lib-modules\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280892 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92432074-635b-4c94-a0b9-363d83ed7127-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280911 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-run-netns\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280918 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4km5m\" (UniqueName: \"kubernetes.io/projected/ea558fad-42cc-435d-aeff-cdb5ada41261-kube-api-access-4km5m\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280942 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea558fad-42cc-435d-aeff-cdb5ada41261-cni-binary-copy\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280966 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-run-multus-certs\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.280987 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c09a0b2-4116-49e0-9727-aa0de72a0b22-dbus\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281010 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-log-socket\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281140 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-log-socket\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281152 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-cni-netd\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281187 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-kubelet\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281202 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-var-lib-openvswitch\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281244 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-systemd-units\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281271 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281273 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-run-ovn-kubernetes\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-sysctl-conf\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281417 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmk4r\" (UniqueName: \"kubernetes.io/projected/0678b753-d3d1-4fd9-950d-eec1f33fcc86-kube-api-access-cmk4r\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281479 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-modprobe-d\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281506 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0678b753-d3d1-4fd9-950d-eec1f33fcc86-ovnkube-config\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-kubernetes\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281540 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92432074-635b-4c94-a0b9-363d83ed7127-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281581 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-var-lib-cni-bin\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0678b753-d3d1-4fd9-950d-eec1f33fcc86-ovnkube-script-lib\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281607 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d998f795-7247-4ec8-ab81-9e15fc38d814-iptables-alerter-script\") pod \"iptables-alerter-fbwxs\" (UID: \"d998f795-7247-4ec8-ab81-9e15fc38d814\") " pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.281910 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d998f795-7247-4ec8-ab81-9e15fc38d814-host-slash\") pod \"iptables-alerter-fbwxs\" (UID: \"d998f795-7247-4ec8-ab81-9e15fc38d814\") " pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281652 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-run-openvswitch\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281684 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-systemd\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281707 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-run-netns\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281726 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-var-lib-kubelet\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281730 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d998f795-7247-4ec8-ab81-9e15fc38d814-host-slash\") pod \"iptables-alerter-fbwxs\" (UID: \"d998f795-7247-4ec8-ab81-9e15fc38d814\") " pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281738 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-run-openvswitch\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281756 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281783 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac-hosts-file\") pod \"node-resolver-7f74m\" (UID: \"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac\") " pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-system-cni-dir\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281856 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-os-release\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281884 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c03f5dc-fe5d-4940-8106-92a596ff68e6-konnectivity-ca\") pod \"konnectivity-agent-xckkk\" (UID: \"9c03f5dc-fe5d-4940-8106-92a596ff68e6\") " pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281953 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-sys\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.281979 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-tuned\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.282466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282014 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c09a0b2-4116-49e0-9727-aa0de72a0b22-kubelet-config\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282054 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0678b753-d3d1-4fd9-950d-eec1f33fcc86-env-overrides\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92432074-635b-4c94-a0b9-363d83ed7127-cni-binary-copy\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282107 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0678b753-d3d1-4fd9-950d-eec1f33fcc86-ovnkube-script-lib\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282148 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-hostroot\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282173 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/627d2b1b-5d4b-416b-ad66-687277f43add-host\") pod \"node-ca-dlvgv\" (UID: \"627d2b1b-5d4b-416b-ad66-687277f43add\") " pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282221 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n5r9\" (UniqueName: \"kubernetes.io/projected/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-kube-api-access-5n5r9\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-cni-dir\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282280 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-conf-dir\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282303 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bbk\" (UniqueName: \"kubernetes.io/projected/627d2b1b-5d4b-416b-ad66-687277f43add-kube-api-access-b5bbk\") pod \"node-ca-dlvgv\" (UID: \"627d2b1b-5d4b-416b-ad66-687277f43add\") " pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282342 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-run-ovn\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282370 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-cni-bin\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-cnibin\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282413 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-run-ovn\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282423 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-cnibin\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282441 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0678b753-d3d1-4fd9-950d-eec1f33fcc86-env-overrides\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282451 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac-tmp-dir\") pod \"node-resolver-7f74m\" (UID: \"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac\") " pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282471 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-cni-bin\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282479 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0678b753-d3d1-4fd9-950d-eec1f33fcc86-ovn-node-metrics-cert\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282509 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-slash\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282534 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-run-systemd\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-node-log\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-sysconfig\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282621 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-host\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282643 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-etc-kubernetes\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282667 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-daemon-config\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282695 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmhv\" (UniqueName: \"kubernetes.io/projected/0de0f376-d574-41e4-a21c-56c2fde4b5d8-kube-api-access-4tmhv\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282726 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8zb\" (UniqueName: \"kubernetes.io/projected/ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac-kube-api-access-nr8zb\") pod \"node-resolver-7f74m\" (UID: \"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac\") " pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.283837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-etc-openvswitch\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282775 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qns2\" (UniqueName: \"kubernetes.io/projected/92432074-635b-4c94-a0b9-363d83ed7127-kube-api-access-7qns2\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282802 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-socket-dir-parent\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-var-lib-cni-multus\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282890 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-host-slash\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282890 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-sys-fs\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282955 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/627d2b1b-5d4b-416b-ad66-687277f43add-serviceca\") pod \"node-ca-dlvgv\" (UID: \"627d2b1b-5d4b-416b-ad66-687277f43add\") " pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.282994 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-system-cni-dir\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.283023 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.283038 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-etc-openvswitch\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.283068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c03f5dc-fe5d-4940-8106-92a596ff68e6-agent-certs\") pod \"konnectivity-agent-xckkk\" (UID: \"9c03f5dc-fe5d-4940-8106-92a596ff68e6\") " pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.283093 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-run\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.283140 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-tmp\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.283163 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.284506 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.283191 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:35.285182 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.283196 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-device-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.285182 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.283236 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-node-log\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.285182 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.283244 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0678b753-d3d1-4fd9-950d-eec1f33fcc86-run-systemd\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.285182 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.283326 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs podName:f115fbf4-80c0-4991-b663-61ce83d6e968 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:35.783291838 +0000 UTC m=+2.991349861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs") pod "network-metrics-daemon-jw459" (UID: "f115fbf4-80c0-4991-b663-61ce83d6e968") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:35.290114 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.290095 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0678b753-d3d1-4fd9-950d-eec1f33fcc86-ovn-node-metrics-cert\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.295350 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.295330 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:56:35.295453 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.295360 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:56:35.295453 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.295374 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7dwqn for pod openshift-network-diagnostics/network-check-target-mvn5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:35.295453 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.295428 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn podName:0c6588bc-27c9-491e-aac7-b6b1597fdfe0 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:35.795409928 +0000 UTC m=+3.003467952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7dwqn" (UniqueName: "kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn") pod "network-check-target-mvn5b" (UID: "0c6588bc-27c9-491e-aac7-b6b1597fdfe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:35.296528 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.296479 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95kgh\" (UniqueName: \"kubernetes.io/projected/f115fbf4-80c0-4991-b663-61ce83d6e968-kube-api-access-95kgh\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:35.296897 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.296856 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8w8p\" (UniqueName: \"kubernetes.io/projected/d998f795-7247-4ec8-ab81-9e15fc38d814-kube-api-access-n8w8p\") pod \"iptables-alerter-fbwxs\" (UID: \"d998f795-7247-4ec8-ab81-9e15fc38d814\") " pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.300072 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.299920 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmk4r\" (UniqueName: \"kubernetes.io/projected/0678b753-d3d1-4fd9-950d-eec1f33fcc86-kube-api-access-cmk4r\") pod \"ovnkube-node-lw2hn\" (UID: \"0678b753-d3d1-4fd9-950d-eec1f33fcc86\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.316401 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.316357 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal" event={"ID":"952eb1cd57023c1b0bde0ddde055ed58","Type":"ContainerStarted","Data":"5aba332f8897cb26ee70ee45a12ceeb730900c29a66cbef91da0d192e47f76ee"} Jul 07 14:56:35.317904 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.317861 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" event={"ID":"d2a0f84d43a03da466d44795084f74c7","Type":"ContainerStarted","Data":"99fd9fdd2923b246793e612d3c0b9e773588d8de187093b1c58fa382892f446c"} Jul 07 14:56:35.384002 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.383968 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-host\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.384110 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384008 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-etc-kubernetes\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.384110 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-daemon-config\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.384110 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384074 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmhv\" (UniqueName: \"kubernetes.io/projected/0de0f376-d574-41e4-a21c-56c2fde4b5d8-kube-api-access-4tmhv\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.384110 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8zb\" (UniqueName: \"kubernetes.io/projected/ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac-kube-api-access-nr8zb\") pod \"node-resolver-7f74m\" (UID: \"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac\") " pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.384281 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384127 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qns2\" (UniqueName: \"kubernetes.io/projected/92432074-635b-4c94-a0b9-363d83ed7127-kube-api-access-7qns2\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.384281 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384157 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-socket-dir-parent\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.384281 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384181 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-var-lib-cni-multus\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.384281 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384203 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-sys-fs\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.384281 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384226 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/627d2b1b-5d4b-416b-ad66-687277f43add-serviceca\") pod \"node-ca-dlvgv\" (UID: \"627d2b1b-5d4b-416b-ad66-687277f43add\") " pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.384281 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-system-cni-dir\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384282 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c03f5dc-fe5d-4940-8106-92a596ff68e6-agent-certs\") pod \"konnectivity-agent-xckkk\" (UID: \"9c03f5dc-fe5d-4940-8106-92a596ff68e6\") " pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384305 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-run\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384326 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-tmp\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384345 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-etc-kubernetes\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384347 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384394 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-device-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384408 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384434 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-sysctl-d\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384436 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-host\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384458 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-registration-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384459 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-socket-dir-parent\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384496 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-var-lib-cni-multus\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384506 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-device-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.384573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384511 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-var-lib-kubelet\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384546 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384570 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-run-k8s-cni-cncf-io\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384575 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-registration-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-os-release\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384603 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-sysctl-d\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384617 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-socket-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-lib-modules\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92432074-635b-4c94-a0b9-363d83ed7127-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384691 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4km5m\" (UniqueName: \"kubernetes.io/projected/ea558fad-42cc-435d-aeff-cdb5ada41261-kube-api-access-4km5m\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384704 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384717 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea558fad-42cc-435d-aeff-cdb5ada41261-cni-binary-copy\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384740 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-var-lib-kubelet\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384800 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-socket-dir\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384739 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-run-multus-certs\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c09a0b2-4116-49e0-9727-aa0de72a0b22-dbus\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384843 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384847 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-sys-fs\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-sysctl-conf\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384859 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-lib-modules\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-modprobe-d\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384916 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-kubernetes\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384921 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-os-release\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384642 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-run-k8s-cni-cncf-io\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384935 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-sysctl-conf\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384941 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92432074-635b-4c94-a0b9-363d83ed7127-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384961 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-run-multus-certs\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384968 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-var-lib-cni-bin\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384994 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-daemon-config\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.385848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.384995 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-systemd\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385034 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c09a0b2-4116-49e0-9727-aa0de72a0b22-dbus\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385038 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-run-netns\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385033 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-systemd\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385085 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-var-lib-kubelet\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385110 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385108 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-system-cni-dir\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.385140 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.385191 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret podName:9c09a0b2-4116-49e0-9727-aa0de72a0b22 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:35.885175125 +0000 UTC m=+3.093233137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret") pod "global-pull-secret-syncer-kwswc" (UID: "9c09a0b2-4116-49e0-9727-aa0de72a0b22") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385197 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0de0f376-d574-41e4-a21c-56c2fde4b5d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385271 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-modprobe-d\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385313 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-kubernetes\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385417 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-var-lib-cni-bin\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385459 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-run-netns\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.386429 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385461 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92432074-635b-4c94-a0b9-363d83ed7127-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385489 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac-hosts-file\") pod \"node-resolver-7f74m\" (UID: \"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac\") " pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385518 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-system-cni-dir\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385544 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-os-release\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c03f5dc-fe5d-4940-8106-92a596ff68e6-konnectivity-ca\") pod \"konnectivity-agent-xckkk\" (UID: \"9c03f5dc-fe5d-4940-8106-92a596ff68e6\") " pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-sys\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385619 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-tuned\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385641 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c09a0b2-4116-49e0-9727-aa0de72a0b22-kubelet-config\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92432074-635b-4c94-a0b9-363d83ed7127-cni-binary-copy\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385731 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92432074-635b-4c94-a0b9-363d83ed7127-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385775 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/627d2b1b-5d4b-416b-ad66-687277f43add-serviceca\") pod \"node-ca-dlvgv\" (UID: \"627d2b1b-5d4b-416b-ad66-687277f43add\") " pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385831 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-host-var-lib-kubelet\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385872 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-system-cni-dir\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.385921 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac-hosts-file\") pod \"node-resolver-7f74m\" (UID: \"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac\") " pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.387067 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386143 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92432074-635b-4c94-a0b9-363d83ed7127-cni-binary-copy\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386223 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c09a0b2-4116-49e0-9727-aa0de72a0b22-kubelet-config\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386239 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c03f5dc-fe5d-4940-8106-92a596ff68e6-konnectivity-ca\") pod \"konnectivity-agent-xckkk\" (UID: \"9c03f5dc-fe5d-4940-8106-92a596ff68e6\") " pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386272 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-sys\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386303 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-hostroot\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386310 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-os-release\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386327 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/627d2b1b-5d4b-416b-ad66-687277f43add-host\") pod \"node-ca-dlvgv\" (UID: \"627d2b1b-5d4b-416b-ad66-687277f43add\") " pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386352 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5r9\" (UniqueName: \"kubernetes.io/projected/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-kube-api-access-5n5r9\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386357 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-run\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386374 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-cni-dir\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386399 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/627d2b1b-5d4b-416b-ad66-687277f43add-host\") pod \"node-ca-dlvgv\" (UID: \"627d2b1b-5d4b-416b-ad66-687277f43add\") " pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386399 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-conf-dir\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386433 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-conf-dir\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386436 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bbk\" (UniqueName: \"kubernetes.io/projected/627d2b1b-5d4b-416b-ad66-687277f43add-kube-api-access-b5bbk\") pod \"node-ca-dlvgv\" (UID: \"627d2b1b-5d4b-416b-ad66-687277f43add\") " pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-cnibin\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.387672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386497 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-cnibin\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac-tmp-dir\") pod \"node-resolver-7f74m\" (UID: \"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac\") " pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386558 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-sysconfig\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386627 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-sysconfig\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386639 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-hostroot\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386687 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92432074-635b-4c94-a0b9-363d83ed7127-cnibin\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386693 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-cnibin\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386704 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea558fad-42cc-435d-aeff-cdb5ada41261-cni-binary-copy\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386869 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea558fad-42cc-435d-aeff-cdb5ada41261-multus-cni-dir\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.386958 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac-tmp-dir\") pod \"node-resolver-7f74m\" (UID: \"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac\") " pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.388367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.387597 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c03f5dc-fe5d-4940-8106-92a596ff68e6-agent-certs\") pod \"konnectivity-agent-xckkk\" (UID: \"9c03f5dc-fe5d-4940-8106-92a596ff68e6\") " pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:35.388822 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.388472 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-tmp\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.388822 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.388609 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-etc-tuned\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.394597 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.394493 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8zb\" (UniqueName: \"kubernetes.io/projected/ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac-kube-api-access-nr8zb\") pod \"node-resolver-7f74m\" (UID: \"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac\") " pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.395241 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.395165 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmhv\" (UniqueName: \"kubernetes.io/projected/0de0f376-d574-41e4-a21c-56c2fde4b5d8-kube-api-access-4tmhv\") pod \"aws-ebs-csi-driver-node-xwlhg\" (UID: \"0de0f376-d574-41e4-a21c-56c2fde4b5d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.396276 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.396254 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5r9\" (UniqueName: \"kubernetes.io/projected/7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b-kube-api-access-5n5r9\") pod \"tuned-vg6bn\" (UID: \"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b\") " pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.396276 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.396265 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bbk\" (UniqueName: \"kubernetes.io/projected/627d2b1b-5d4b-416b-ad66-687277f43add-kube-api-access-b5bbk\") pod \"node-ca-dlvgv\" (UID: \"627d2b1b-5d4b-416b-ad66-687277f43add\") " pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.396708 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.396689 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qns2\" (UniqueName: \"kubernetes.io/projected/92432074-635b-4c94-a0b9-363d83ed7127-kube-api-access-7qns2\") pod \"multus-additional-cni-plugins-mn7mm\" (UID: \"92432074-635b-4c94-a0b9-363d83ed7127\") " pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.397537 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.397519 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4km5m\" (UniqueName: \"kubernetes.io/projected/ea558fad-42cc-435d-aeff-cdb5ada41261-kube-api-access-4km5m\") pod \"multus-cfcxp\" (UID: \"ea558fad-42cc-435d-aeff-cdb5ada41261\") " pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.484880 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.484854 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbwxs" Jul 07 14:56:35.494986 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.494966 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:56:35.506713 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.506688 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:35.512227 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.512208 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" Jul 07 14:56:35.519854 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.519834 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" Jul 07 14:56:35.533595 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.533543 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f74m" Jul 07 14:56:35.541223 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.541205 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dlvgv" Jul 07 14:56:35.548837 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.548819 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" Jul 07 14:56:35.556591 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.556571 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cfcxp" Jul 07 14:56:35.790207 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.790137 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:35.790338 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.790293 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:35.790376 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.790358 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs podName:f115fbf4-80c0-4991-b663-61ce83d6e968 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:36.790344513 +0000 UTC m=+3.998402526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs") pod "network-metrics-daemon-jw459" (UID: "f115fbf4-80c0-4991-b663-61ce83d6e968") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:35.890688 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.890648 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:35.890852 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:35.890707 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:35.890852 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.890806 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:35.890852 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.890825 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:56:35.890852 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.890845 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:56:35.890852 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.890857 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7dwqn for pod openshift-network-diagnostics/network-check-target-mvn5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:35.891066 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.890875 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret podName:9c09a0b2-4116-49e0-9727-aa0de72a0b22 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:36.89086071 +0000 UTC m=+4.098918722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret") pod "global-pull-secret-syncer-kwswc" (UID: "9c09a0b2-4116-49e0-9727-aa0de72a0b22") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:35.891066 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:35.890907 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn podName:0c6588bc-27c9-491e-aac7-b6b1597fdfe0 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:36.890891955 +0000 UTC m=+4.098949964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7dwqn" (UniqueName: "kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn") pod "network-check-target-mvn5b" (UID: "0c6588bc-27c9-491e-aac7-b6b1597fdfe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:36.214990 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:36.214894 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:51:35 +0000 UTC" deadline="2028-04-20 05:46:13.147147536 +0000 UTC" Jul 07 14:56:36.214990 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:36.214930 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15662h49m36.932221303s" Jul 07 14:56:36.305690 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:36.305654 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:36.306257 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:36.305804 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:36.436674 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:36.436641 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92432074_635b_4c94_a0b9_363d83ed7127.slice/crio-a339d4d9c03db9bd02489effc87d0464e84d5d61e0c074173c3436f1920ee4c6 WatchSource:0}: Error finding container a339d4d9c03db9bd02489effc87d0464e84d5d61e0c074173c3436f1920ee4c6: Status 404 returned error can't find the container with id a339d4d9c03db9bd02489effc87d0464e84d5d61e0c074173c3436f1920ee4c6 Jul 07 14:56:36.437960 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:36.437940 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c03f5dc_fe5d_4940_8106_92a596ff68e6.slice/crio-f57aed42a477c82278e566e7eb47217659d2af1fed8c464f8d20e98bc2cb1f89 WatchSource:0}: Error finding container f57aed42a477c82278e566e7eb47217659d2af1fed8c464f8d20e98bc2cb1f89: Status 404 returned error can't find the container with id f57aed42a477c82278e566e7eb47217659d2af1fed8c464f8d20e98bc2cb1f89 Jul 07 14:56:36.441880 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:36.441855 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb0a2d9_e3b4_43a6_aa29_e0e8d597da4b.slice/crio-d6fe8b397cc3faa24871097219b5c6b6980367d40273694fd6d7a6160993272c WatchSource:0}: Error finding container d6fe8b397cc3faa24871097219b5c6b6980367d40273694fd6d7a6160993272c: Status 404 returned error can't find the container with id d6fe8b397cc3faa24871097219b5c6b6980367d40273694fd6d7a6160993272c Jul 07 14:56:36.443218 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:36.443197 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627d2b1b_5d4b_416b_ad66_687277f43add.slice/crio-b5b5d3d76d06dff4f4c600ff2d72b2028d14bb577a96ceaa7a9157f40c9046d8 WatchSource:0}: Error finding container b5b5d3d76d06dff4f4c600ff2d72b2028d14bb577a96ceaa7a9157f40c9046d8: Status 404 returned error can't find the container with id b5b5d3d76d06dff4f4c600ff2d72b2028d14bb577a96ceaa7a9157f40c9046d8 Jul 07 14:56:36.443648 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:36.443619 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd998f795_7247_4ec8_ab81_9e15fc38d814.slice/crio-e813eade98f69a60fd1106294bdb62f931b94c429c712c99e1302e6bd261b08b WatchSource:0}: Error finding container e813eade98f69a60fd1106294bdb62f931b94c429c712c99e1302e6bd261b08b: Status 404 returned error can't find the container with id e813eade98f69a60fd1106294bdb62f931b94c429c712c99e1302e6bd261b08b Jul 07 14:56:36.445994 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:56:36.445699 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea558fad_42cc_435d_aeff_cdb5ada41261.slice/crio-5f3752bf1a382abdf34e9d734d0f885f7368e481a64eddde761b9b79a71de519 WatchSource:0}: Error finding container 5f3752bf1a382abdf34e9d734d0f885f7368e481a64eddde761b9b79a71de519: Status 404 returned error can't find the container with id 5f3752bf1a382abdf34e9d734d0f885f7368e481a64eddde761b9b79a71de519 Jul 07 14:56:36.798312 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:36.798001 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:36.798312 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:36.798159 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:36.798506 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:36.798351 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs podName:f115fbf4-80c0-4991-b663-61ce83d6e968 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:38.798334387 +0000 UTC m=+6.006392396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs") pod "network-metrics-daemon-jw459" (UID: "f115fbf4-80c0-4991-b663-61ce83d6e968") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:36.899356 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:36.899318 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:36.899515 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:36.899363 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:36.899515 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:36.899475 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:36.899515 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:36.899490 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:56:36.899682 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:36.899522 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:56:36.899682 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:36.899536 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7dwqn for pod openshift-network-diagnostics/network-check-target-mvn5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:36.899682 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:36.899541 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret podName:9c09a0b2-4116-49e0-9727-aa0de72a0b22 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:38.899522969 +0000 UTC m=+6.107580997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret") pod "global-pull-secret-syncer-kwswc" (UID: "9c09a0b2-4116-49e0-9727-aa0de72a0b22") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:36.899682 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:36.899585 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn podName:0c6588bc-27c9-491e-aac7-b6b1597fdfe0 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:38.899574798 +0000 UTC m=+6.107632822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7dwqn" (UniqueName: "kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn") pod "network-check-target-mvn5b" (UID: "0c6588bc-27c9-491e-aac7-b6b1597fdfe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:37.216077 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.215942 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:51:35 +0000 UTC" deadline="2028-04-19 22:05:10.20019007 +0000 UTC" Jul 07 14:56:37.216077 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.216006 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15655h8m32.984188933s" Jul 07 14:56:37.310716 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.310578 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:37.310716 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:37.310702 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:37.311333 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.310775 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:37.311333 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:37.310837 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:37.344740 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.344701 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal" event={"ID":"952eb1cd57023c1b0bde0ddde055ed58","Type":"ContainerStarted","Data":"77d045b714916356fdd03641f09cc0f146081a1f4d6485a8f0932c7ddcf210e7"} Jul 07 14:56:37.362880 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.362807 2566 generic.go:356] "Generic (PLEG): container finished" podID="d2a0f84d43a03da466d44795084f74c7" containerID="faf468f2ba2286d14c69a2ee8551a35d7c46406c76fb09ec2aaba3432e27322b" exitCode=0 Jul 07 14:56:37.363240 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.363121 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" event={"ID":"d2a0f84d43a03da466d44795084f74c7","Type":"ContainerDied","Data":"faf468f2ba2286d14c69a2ee8551a35d7c46406c76fb09ec2aaba3432e27322b"} Jul 07 14:56:37.374468 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.374194 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" event={"ID":"0678b753-d3d1-4fd9-950d-eec1f33fcc86","Type":"ContainerStarted","Data":"8d934fb4a68a618e451e1fd6fb17d810f88bde3375a56f45a20373f392889229"} Jul 07 14:56:37.391914 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.391422 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-13.ec2.internal" podStartSLOduration=3.391398689 podStartE2EDuration="3.391398689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:37.370987373 +0000 UTC m=+4.579045404" watchObservedRunningTime="2026-07-07 14:56:37.391398689 +0000 UTC m=+4.599456725" Jul 07 14:56:37.398964 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.398864 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dlvgv" event={"ID":"627d2b1b-5d4b-416b-ad66-687277f43add","Type":"ContainerStarted","Data":"b5b5d3d76d06dff4f4c600ff2d72b2028d14bb577a96ceaa7a9157f40c9046d8"} Jul 07 14:56:37.406823 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.406732 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f74m" event={"ID":"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac","Type":"ContainerStarted","Data":"3fac3a56176d93cbedf82de3c953dba5cd4aac1e12245e9cdd2c1aa487753409"} Jul 07 14:56:37.417163 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.417086 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" event={"ID":"0de0f376-d574-41e4-a21c-56c2fde4b5d8","Type":"ContainerStarted","Data":"730c55a1a33c062a7635f1b9b74e5a9d2ba0b6cea14282e444708d0c66fba850"} Jul 07 14:56:37.433974 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.433874 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cfcxp" event={"ID":"ea558fad-42cc-435d-aeff-cdb5ada41261","Type":"ContainerStarted","Data":"5f3752bf1a382abdf34e9d734d0f885f7368e481a64eddde761b9b79a71de519"} Jul 07 14:56:37.454460 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.454410 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbwxs" event={"ID":"d998f795-7247-4ec8-ab81-9e15fc38d814","Type":"ContainerStarted","Data":"e813eade98f69a60fd1106294bdb62f931b94c429c712c99e1302e6bd261b08b"} Jul 07 14:56:37.466387 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.466229 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" event={"ID":"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b","Type":"ContainerStarted","Data":"d6fe8b397cc3faa24871097219b5c6b6980367d40273694fd6d7a6160993272c"} Jul 07 14:56:37.473861 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.473825 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xckkk" event={"ID":"9c03f5dc-fe5d-4940-8106-92a596ff68e6","Type":"ContainerStarted","Data":"f57aed42a477c82278e566e7eb47217659d2af1fed8c464f8d20e98bc2cb1f89"} Jul 07 14:56:37.481869 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:37.481832 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" event={"ID":"92432074-635b-4c94-a0b9-363d83ed7127","Type":"ContainerStarted","Data":"a339d4d9c03db9bd02489effc87d0464e84d5d61e0c074173c3436f1920ee4c6"} Jul 07 14:56:38.305688 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:38.305611 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:38.305847 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:38.305757 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:38.493179 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:38.493108 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" event={"ID":"d2a0f84d43a03da466d44795084f74c7","Type":"ContainerStarted","Data":"af1205c620cce23337493441477564921e47b18fa98b57d6b601dfa6c6bb5408"} Jul 07 14:56:38.817275 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:38.816667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:38.817275 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:38.816822 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:38.817275 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:38.816895 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs podName:f115fbf4-80c0-4991-b663-61ce83d6e968 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:42.816876202 +0000 UTC m=+10.024934233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs") pod "network-metrics-daemon-jw459" (UID: "f115fbf4-80c0-4991-b663-61ce83d6e968") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:38.917329 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:38.917288 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:38.917519 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:38.917355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:38.917576 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:38.917525 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:38.917622 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:38.917590 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret podName:9c09a0b2-4116-49e0-9727-aa0de72a0b22 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:42.917571063 +0000 UTC m=+10.125629087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret") pod "global-pull-secret-syncer-kwswc" (UID: "9c09a0b2-4116-49e0-9727-aa0de72a0b22") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:38.918118 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:38.918030 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:56:38.918118 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:38.918069 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:56:38.918118 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:38.918083 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7dwqn for pod openshift-network-diagnostics/network-check-target-mvn5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:38.918335 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:38.918128 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn podName:0c6588bc-27c9-491e-aac7-b6b1597fdfe0 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:42.918113399 +0000 UTC m=+10.126171411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7dwqn" (UniqueName: "kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn") pod "network-check-target-mvn5b" (UID: "0c6588bc-27c9-491e-aac7-b6b1597fdfe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:39.310930 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:39.310845 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:39.311149 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:39.311095 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:39.311390 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:39.311354 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:39.311511 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:39.311493 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:40.306378 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:40.306313 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:40.306820 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:40.306478 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:41.306085 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:41.306056 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:41.306232 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:41.306181 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:41.306772 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:41.306626 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:41.306772 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:41.306728 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:42.306524 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:42.306479 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:42.306707 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:42.306652 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:42.853590 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:42.853558 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:42.854031 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:42.853678 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:42.854031 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:42.853747 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs podName:f115fbf4-80c0-4991-b663-61ce83d6e968 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:50.853726495 +0000 UTC m=+18.061784506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs") pod "network-metrics-daemon-jw459" (UID: "f115fbf4-80c0-4991-b663-61ce83d6e968") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:42.954227 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:42.954191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:42.954431 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:42.954245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:42.954431 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:42.954387 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:42.954431 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:42.954392 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:56:42.954431 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:42.954420 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:56:42.954626 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:42.954441 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret podName:9c09a0b2-4116-49e0-9727-aa0de72a0b22 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:50.954422618 +0000 UTC m=+18.162480629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret") pod "global-pull-secret-syncer-kwswc" (UID: "9c09a0b2-4116-49e0-9727-aa0de72a0b22") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:42.954626 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:42.954445 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7dwqn for pod openshift-network-diagnostics/network-check-target-mvn5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:42.954626 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:42.954496 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn podName:0c6588bc-27c9-491e-aac7-b6b1597fdfe0 nodeName:}" failed. No retries permitted until 2026-07-07 14:56:50.95447867 +0000 UTC m=+18.162536685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7dwqn" (UniqueName: "kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn") pod "network-check-target-mvn5b" (UID: "0c6588bc-27c9-491e-aac7-b6b1597fdfe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:43.307143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:43.307061 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:43.307296 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:43.307185 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:43.307574 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:43.307539 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:43.307707 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:43.307654 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:44.306327 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:44.306296 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:44.306711 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:44.306443 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:45.309913 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:45.309885 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:45.310448 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:45.310000 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:45.310448 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:45.310070 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:45.310448 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:45.310186 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:46.305792 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:46.305759 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:46.305966 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:46.305891 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:47.306335 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:47.306296 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:47.306794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:47.306296 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:47.306794 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:47.306440 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:47.306794 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:47.306513 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:48.306423 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:48.306389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:48.306862 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:48.306513 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:49.306662 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:49.306628 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:49.307073 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:49.306677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:49.307073 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:49.306767 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:49.307073 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:49.306908 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:50.306438 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:50.306402 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:50.306644 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:50.306547 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:50.916002 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:50.915970 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:50.916484 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:50.916123 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:50.916484 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:50.916184 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs podName:f115fbf4-80c0-4991-b663-61ce83d6e968 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:06.916166019 +0000 UTC m=+34.124224030 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs") pod "network-metrics-daemon-jw459" (UID: "f115fbf4-80c0-4991-b663-61ce83d6e968") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:56:51.016608 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:51.016573 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:51.016792 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:51.016622 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:51.016792 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:51.016753 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:51.016792 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:51.016775 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:56:51.016947 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:51.016803 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:56:51.016947 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:51.016816 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret podName:9c09a0b2-4116-49e0-9727-aa0de72a0b22 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:07.016802494 +0000 UTC m=+34.224860504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret") pod "global-pull-secret-syncer-kwswc" (UID: "9c09a0b2-4116-49e0-9727-aa0de72a0b22") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:56:51.016947 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:51.016817 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7dwqn for pod openshift-network-diagnostics/network-check-target-mvn5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:51.016947 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:51.016856 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn podName:0c6588bc-27c9-491e-aac7-b6b1597fdfe0 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:07.016847189 +0000 UTC m=+34.224905202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7dwqn" (UniqueName: "kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn") pod "network-check-target-mvn5b" (UID: "0c6588bc-27c9-491e-aac7-b6b1597fdfe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:56:51.308874 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:51.308798 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:51.309030 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:51.308798 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:51.309030 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:51.308908 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:51.309030 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:51.308965 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:52.306275 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:52.306237 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:52.306701 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:52.306362 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:53.307247 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:53.307210 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:53.307687 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:53.307326 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:53.307687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:53.307379 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:53.307687 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:53.307485 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:54.306146 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:54.306080 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:54.306315 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:54.306237 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:55.308982 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.308811 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:55.309687 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.308861 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:55.309687 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:55.309079 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:55.309687 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:55.309145 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:55.527620 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.527595 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" event={"ID":"0678b753-d3d1-4fd9-950d-eec1f33fcc86","Type":"ContainerStarted","Data":"d37f9c8f929398027b4a2e9afaeda35f71c065cd726f44bd34fd3f8e5dbf7a78"} Jul 07 14:56:55.527812 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.527662 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" event={"ID":"0678b753-d3d1-4fd9-950d-eec1f33fcc86","Type":"ContainerStarted","Data":"0f45bf001b8a2d848d55711be615f72aeca10616a1b965358fdb88059fb06aaa"} Jul 07 14:56:55.527812 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.527693 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" event={"ID":"0678b753-d3d1-4fd9-950d-eec1f33fcc86","Type":"ContainerStarted","Data":"4ec82e6e670a86a129cf11efaa6bc5a6967d21b79fc8eb718c11663ac6cd6580"} Jul 07 14:56:55.527812 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.527721 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" event={"ID":"0678b753-d3d1-4fd9-950d-eec1f33fcc86","Type":"ContainerStarted","Data":"6f921c428710eedd43527fa68ef5e9412594a1efbcec80e3ee126b5cfa6714e9"} Jul 07 14:56:55.527812 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.527747 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" event={"ID":"0678b753-d3d1-4fd9-950d-eec1f33fcc86","Type":"ContainerStarted","Data":"748684f0357e0f3dfd92a599a2d2f10146244ac02e8d3d02128498b4e7b62efe"} Jul 07 14:56:55.527812 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.527774 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" event={"ID":"0678b753-d3d1-4fd9-950d-eec1f33fcc86","Type":"ContainerStarted","Data":"edcf85f1f3a46115712e12262ab1c8630ddf57976102ff2627e8dd205fc1375d"} Jul 07 14:56:55.528942 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.528914 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dlvgv" event={"ID":"627d2b1b-5d4b-416b-ad66-687277f43add","Type":"ContainerStarted","Data":"208753d54ffc555f23713d409887f063252e4ee504eba64f5f2f1d78b9ceb0bf"} Jul 07 14:56:55.530252 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.530224 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f74m" event={"ID":"ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac","Type":"ContainerStarted","Data":"a9b92fb896d9c7541295d11e08a6b05a571abfc0df097abc813103e0ad11f824"} Jul 07 14:56:55.531697 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.531671 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" event={"ID":"0de0f376-d574-41e4-a21c-56c2fde4b5d8","Type":"ContainerStarted","Data":"8aa6c43e303e5a3521b30ec8805ba83c1c0b270d3e10a302c6fb105855c07155"} Jul 07 14:56:55.533174 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.533150 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cfcxp" event={"ID":"ea558fad-42cc-435d-aeff-cdb5ada41261","Type":"ContainerStarted","Data":"118ef7b059588d1c59fcfcd68c725510c440e98437c68d295c46f2f705007546"} Jul 07 14:56:55.534617 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.534598 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" event={"ID":"7fb0a2d9-e3b4-43a6-aa29-e0e8d597da4b","Type":"ContainerStarted","Data":"30f820140a0ab4809efb31a761a96d5f7ad56734d5019da65c92f2984cf57e05"} Jul 07 14:56:55.535978 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.535958 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xckkk" event={"ID":"9c03f5dc-fe5d-4940-8106-92a596ff68e6","Type":"ContainerStarted","Data":"338db3d98e007ee3002530526777c849276fb435c79583104a8a82843ef7f71e"} Jul 07 14:56:55.537373 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.537352 2566 generic.go:356] "Generic (PLEG): container finished" podID="92432074-635b-4c94-a0b9-363d83ed7127" containerID="4821287f2add287d0f1a293205484fc1b850a91aa0de781126451afa22ce70c9" exitCode=0 Jul 07 14:56:55.537468 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.537382 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" event={"ID":"92432074-635b-4c94-a0b9-363d83ed7127","Type":"ContainerDied","Data":"4821287f2add287d0f1a293205484fc1b850a91aa0de781126451afa22ce70c9"} Jul 07 14:56:55.549354 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.549314 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-13.ec2.internal" podStartSLOduration=21.549286966 podStartE2EDuration="21.549286966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:38.514855926 +0000 UTC m=+5.722913954" watchObservedRunningTime="2026-07-07 14:56:55.549286966 +0000 UTC m=+22.757344981" Jul 07 14:56:55.549819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.549799 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dlvgv" podStartSLOduration=8.601017992 podStartE2EDuration="22.549791375s" totalImagesPullingTime="13.948773383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:55.549070301 +0000 UTC m=+22.757128332" watchObservedRunningTime="2026-07-07 14:56:55.549791375 +0000 UTC m=+22.757849406" Jul 07 14:56:55.567991 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.567918 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vg6bn" podStartSLOduration=4.206112139 podStartE2EDuration="22.567904447s" totalImagesPullingTime="18.361792308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:55.567699835 +0000 UTC m=+22.775757867" watchObservedRunningTime="2026-07-07 14:56:55.567904447 +0000 UTC m=+22.775962479" Jul 07 14:56:55.615661 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.615619 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xckkk" podStartSLOduration=8.661181991 podStartE2EDuration="22.615604836s" totalImagesPullingTime="13.954422845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:55.615324211 +0000 UTC m=+22.823382242" watchObservedRunningTime="2026-07-07 14:56:55.615604836 +0000 UTC m=+22.823662867" Jul 07 14:56:55.615817 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.615792 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7f74m" podStartSLOduration=4.261886912 podStartE2EDuration="22.615788699s" totalImagesPullingTime="18.353901787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:55.58836916 +0000 UTC m=+22.796427193" watchObservedRunningTime="2026-07-07 14:56:55.615788699 +0000 UTC m=+22.823846730" Jul 07 14:56:55.659630 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:55.659580 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cfcxp" podStartSLOduration=4.25380702 podStartE2EDuration="22.659563719s" totalImagesPullingTime="18.405756699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:55.659274325 +0000 UTC m=+22.867332358" watchObservedRunningTime="2026-07-07 14:56:55.659563719 +0000 UTC m=+22.867621750" Jul 07 14:56:56.305994 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:56.305951 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:56.306230 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:56.306112 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:56.528179 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:56.528145 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 14:56:56.542193 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:56.542158 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" event={"ID":"0de0f376-d574-41e4-a21c-56c2fde4b5d8","Type":"ContainerStarted","Data":"ffd5b2f31acefa8d88bb94dad4a33c79a4f97bef66057dc1566a92f2a9f4f0d7"} Jul 07 14:56:56.543936 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:56.543899 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbwxs" event={"ID":"d998f795-7247-4ec8-ab81-9e15fc38d814","Type":"ContainerStarted","Data":"e39d1e917e2ff4ce53dfc670a6ff372218f7f332c5d789ba4b1b2a101977d870"} Jul 07 14:56:56.560783 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:56.560735 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fbwxs" podStartSLOduration=9.61197593 podStartE2EDuration="23.56071795s" totalImagesPullingTime="13.94874202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:56.560577784 +0000 UTC m=+23.768635817" watchObservedRunningTime="2026-07-07 14:56:56.56071795 +0000 UTC m=+23.768775982" Jul 07 14:56:57.249153 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:57.249055 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:56:56.528168364Z","UUID":"dd99b542-bf84-4d95-8a95-5d60baaa9b07","Handler":null,"Name":"","Endpoint":""} Jul 07 14:56:57.250551 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:57.250532 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 14:56:57.250651 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:57.250567 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 14:56:57.306550 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:57.306512 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:57.306550 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:57.306551 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:57.306769 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:57.306647 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:56:57.306769 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:57.306725 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:58.278563 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:58.278529 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:58.279121 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:58.279097 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:58.306160 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:58.306142 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:56:58.306268 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:58.306247 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:56:58.552600 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:58.552513 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" event={"ID":"0678b753-d3d1-4fd9-950d-eec1f33fcc86","Type":"ContainerStarted","Data":"30985eadb80e315c4ec2fc80ed499b23b73629bc7b0214cb0154663beb6c90f8"} Jul 07 14:56:58.554782 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:58.554751 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" event={"ID":"0de0f376-d574-41e4-a21c-56c2fde4b5d8","Type":"ContainerStarted","Data":"5e967504bed44a780915f0fb38c5e34c17f154b4621fe5da0f70006c9c644f79"} Jul 07 14:56:58.555473 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:58.555127 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:58.555653 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:58.555629 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xckkk" Jul 07 14:56:58.580481 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:58.580434 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xwlhg" podStartSLOduration=4.230905421 podStartE2EDuration="25.580413088s" totalImagesPullingTime="21.349507667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:58.579861075 +0000 UTC m=+25.787919145" watchObservedRunningTime="2026-07-07 14:56:58.580413088 +0000 UTC m=+25.788471120" Jul 07 14:56:59.306726 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:59.306531 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:56:59.307175 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:56:59.306595 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:56:59.307175 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:59.306819 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:56:59.307175 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:56:59.306863 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:57:00.305758 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:00.305676 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:57:00.305902 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:00.305827 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:57:01.306347 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:01.306258 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:01.306347 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:01.306301 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:01.306988 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:01.306392 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:57:01.306988 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:01.306541 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:57:01.566587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:01.566469 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" event={"ID":"0678b753-d3d1-4fd9-950d-eec1f33fcc86","Type":"ContainerStarted","Data":"50c418d074521bf5e09138cd85cfd12e8e67b1d3fb3368d7699f21dc6e22f08d"} Jul 07 14:57:01.566883 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:01.566737 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:57:01.582669 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:01.582644 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:57:01.611032 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:01.610831 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" podStartSLOduration=10.195761511 podStartE2EDuration="28.610816087s" totalImagesPullingTime="18.415054576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:01.60308716 +0000 UTC m=+28.811145192" watchObservedRunningTime="2026-07-07 14:57:01.610816087 +0000 UTC m=+28.818874118" Jul 07 14:57:02.306023 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:02.305991 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:57:02.306273 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:02.306114 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:57:02.570737 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:02.570652 2566 generic.go:356] "Generic (PLEG): container finished" podID="92432074-635b-4c94-a0b9-363d83ed7127" containerID="454e286ece44566e2597e4587cf3fe47006c32e8075fdf3734506d01ba00259c" exitCode=0 Jul 07 14:57:02.571367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:02.570743 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" event={"ID":"92432074-635b-4c94-a0b9-363d83ed7127","Type":"ContainerDied","Data":"454e286ece44566e2597e4587cf3fe47006c32e8075fdf3734506d01ba00259c"} Jul 07 14:57:02.571367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:02.571154 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:57:02.571367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:02.571223 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:57:02.586805 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:02.586785 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:57:03.307269 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:03.307233 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:03.307419 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:03.307354 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:57:03.307419 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:03.307403 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:03.307547 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:03.307523 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:57:03.714450 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:03.714378 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kwswc"] Jul 07 14:57:03.714821 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:03.714488 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:03.714821 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:03.714566 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:57:03.718439 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:03.718417 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mvn5b"] Jul 07 14:57:03.718565 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:03.718500 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:03.718626 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:03.718576 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:57:03.719943 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:03.719923 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jw459"] Jul 07 14:57:03.720021 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:03.720013 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:57:03.720131 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:03.720108 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:57:04.577228 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:04.577032 2566 generic.go:356] "Generic (PLEG): container finished" podID="92432074-635b-4c94-a0b9-363d83ed7127" containerID="9432e8f571fd76fed3cdce048c641cbb1082dc9baf3f79ac80da77a03d56e4e4" exitCode=0 Jul 07 14:57:04.577401 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:04.577080 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" event={"ID":"92432074-635b-4c94-a0b9-363d83ed7127","Type":"ContainerDied","Data":"9432e8f571fd76fed3cdce048c641cbb1082dc9baf3f79ac80da77a03d56e4e4"} Jul 07 14:57:05.305977 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:05.305943 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:05.306605 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:05.306001 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:57:05.306605 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:05.306020 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:05.306605 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:05.306102 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:57:05.306605 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:05.306149 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:57:05.306605 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:05.306171 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:57:06.583753 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:06.583717 2566 generic.go:356] "Generic (PLEG): container finished" podID="92432074-635b-4c94-a0b9-363d83ed7127" containerID="035e311f71cba7445ff69e337603a4b7a8d43e6d8ce280b6a5438b2e65cce6ed" exitCode=0 Jul 07 14:57:06.584305 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:06.583780 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" event={"ID":"92432074-635b-4c94-a0b9-363d83ed7127","Type":"ContainerDied","Data":"035e311f71cba7445ff69e337603a4b7a8d43e6d8ce280b6a5438b2e65cce6ed"} Jul 07 14:57:06.933693 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:06.933589 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:57:06.933866 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:06.933755 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:57:06.933866 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:06.933830 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs podName:f115fbf4-80c0-4991-b663-61ce83d6e968 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:38.933810533 +0000 UTC m=+66.141868543 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs") pod "network-metrics-daemon-jw459" (UID: "f115fbf4-80c0-4991-b663-61ce83d6e968") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:57:07.034344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:07.034306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:07.034344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:07.034350 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:07.034546 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:07.034454 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:57:07.034546 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:07.034465 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:57:07.034546 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:07.034482 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:57:07.034546 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:07.034495 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7dwqn for pod openshift-network-diagnostics/network-check-target-mvn5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:57:07.034546 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:07.034511 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret podName:9c09a0b2-4116-49e0-9727-aa0de72a0b22 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:39.034494371 +0000 UTC m=+66.242552384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret") pod "global-pull-secret-syncer-kwswc" (UID: "9c09a0b2-4116-49e0-9727-aa0de72a0b22") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:57:07.034546 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:07.034530 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn podName:0c6588bc-27c9-491e-aac7-b6b1597fdfe0 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:39.034520816 +0000 UTC m=+66.242578841 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7dwqn" (UniqueName: "kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn") pod "network-check-target-mvn5b" (UID: "0c6588bc-27c9-491e-aac7-b6b1597fdfe0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:57:07.306413 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:07.306382 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:07.306593 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:07.306382 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:57:07.306593 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:07.306508 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvn5b" podUID="0c6588bc-27c9-491e-aac7-b6b1597fdfe0" Jul 07 14:57:07.306593 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:07.306382 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:07.306865 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:07.306615 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw459" podUID="f115fbf4-80c0-4991-b663-61ce83d6e968" Jul 07 14:57:07.306865 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:07.306691 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kwswc" podUID="9c09a0b2-4116-49e0-9727-aa0de72a0b22" Jul 07 14:57:08.558590 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.558505 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-13.ec2.internal" event="NodeReady" Jul 07 14:57:08.559103 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.558673 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 14:57:08.604171 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.604142 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-696d4545c5-pc6zb"] Jul 07 14:57:08.605944 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.605918 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.608995 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.608976 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 14:57:08.609357 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.609340 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 14:57:08.609449 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.609340 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 14:57:08.609449 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.609376 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dszn9\"" Jul 07 14:57:08.619121 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.619100 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 14:57:08.622675 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.622651 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-696d4545c5-pc6zb"] Jul 07 14:57:08.631161 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.631137 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lq95g"] Jul 07 14:57:08.632981 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.632961 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nmchs"] Jul 07 14:57:08.633129 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.633112 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.635018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.634987 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nmchs" Jul 07 14:57:08.637184 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.637162 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 14:57:08.637408 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.637395 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9hq6c\"" Jul 07 14:57:08.637676 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.637661 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 14:57:08.638088 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.637894 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 14:57:08.638088 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.637936 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 14:57:08.638088 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.637944 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 14:57:08.638088 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.637958 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tc5rk\"" Jul 07 14:57:08.645203 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.645183 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nmchs"] Jul 07 14:57:08.653199 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.653175 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lq95g"] Jul 07 14:57:08.747644 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.747604 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02236daa-01c6-4fcf-b497-444d8258a20b-ca-trust-extracted\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.747824 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.747646 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5kt\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-kube-api-access-8t5kt\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.747824 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.747676 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.747824 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.747750 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-installation-pull-secrets\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.747824 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.747795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfxpf\" (UniqueName: \"kubernetes.io/projected/a7397841-807c-42d8-b55f-920d4adeb369-kube-api-access-jfxpf\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.748038 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.747834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.748038 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.747859 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtcb\" (UniqueName: \"kubernetes.io/projected/c36255e9-35a1-4c63-8154-aaa712e37e96-kube-api-access-jjtcb\") pod \"ingress-canary-nmchs\" (UID: \"c36255e9-35a1-4c63-8154-aaa712e37e96\") " pod="openshift-ingress-canary/ingress-canary-nmchs" Jul 07 14:57:08.748038 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.747907 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-trusted-ca\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.748038 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.747942 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7397841-807c-42d8-b55f-920d4adeb369-config-volume\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.748038 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.748008 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-bound-sa-token\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.748038 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.748035 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c36255e9-35a1-4c63-8154-aaa712e37e96-cert\") pod \"ingress-canary-nmchs\" (UID: \"c36255e9-35a1-4c63-8154-aaa712e37e96\") " pod="openshift-ingress-canary/ingress-canary-nmchs" Jul 07 14:57:08.748348 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.748108 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-image-registry-private-configuration\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.748348 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.748136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7397841-807c-42d8-b55f-920d4adeb369-tmp-dir\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.748348 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.748209 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-registry-certificates\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.849462 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849380 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-registry-certificates\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.849462 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02236daa-01c6-4fcf-b497-444d8258a20b-ca-trust-extracted\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.849667 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849486 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5kt\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-kube-api-access-8t5kt\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.849667 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849513 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.849667 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849551 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-installation-pull-secrets\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.849667 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849576 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfxpf\" (UniqueName: \"kubernetes.io/projected/a7397841-807c-42d8-b55f-920d4adeb369-kube-api-access-jfxpf\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.849667 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849605 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.849667 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtcb\" (UniqueName: \"kubernetes.io/projected/c36255e9-35a1-4c63-8154-aaa712e37e96-kube-api-access-jjtcb\") pod \"ingress-canary-nmchs\" (UID: \"c36255e9-35a1-4c63-8154-aaa712e37e96\") " pod="openshift-ingress-canary/ingress-canary-nmchs" Jul 07 14:57:08.849667 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-trusted-ca\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.849908 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849692 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7397841-807c-42d8-b55f-920d4adeb369-config-volume\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.849908 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849721 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-bound-sa-token\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.849908 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849739 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c36255e9-35a1-4c63-8154-aaa712e37e96-cert\") pod \"ingress-canary-nmchs\" (UID: \"c36255e9-35a1-4c63-8154-aaa712e37e96\") " pod="openshift-ingress-canary/ingress-canary-nmchs" Jul 07 14:57:08.849908 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849762 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-image-registry-private-configuration\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.849908 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849781 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7397841-807c-42d8-b55f-920d4adeb369-tmp-dir\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.850066 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.849915 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02236daa-01c6-4fcf-b497-444d8258a20b-ca-trust-extracted\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.850066 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.850021 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7397841-807c-42d8-b55f-920d4adeb369-tmp-dir\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.850147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.850106 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-registry-certificates\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.850249 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:08.850226 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:57:08.850321 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:08.850250 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696d4545c5-pc6zb: secret "image-registry-tls" not found Jul 07 14:57:08.850376 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:08.850323 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls podName:02236daa-01c6-4fcf-b497-444d8258a20b nodeName:}" failed. No retries permitted until 2026-07-07 14:57:09.350303662 +0000 UTC m=+36.558361692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls") pod "image-registry-696d4545c5-pc6zb" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b") : secret "image-registry-tls" not found Jul 07 14:57:08.850638 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:08.850606 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:57:08.850762 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:08.850659 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls podName:a7397841-807c-42d8-b55f-920d4adeb369 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:09.350643712 +0000 UTC m=+36.558701740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls") pod "dns-default-lq95g" (UID: "a7397841-807c-42d8-b55f-920d4adeb369") : secret "dns-default-metrics-tls" not found Jul 07 14:57:08.850830 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.850781 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7397841-807c-42d8-b55f-920d4adeb369-config-volume\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.850937 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.850911 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-trusted-ca\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.854567 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.854542 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-image-registry-private-configuration\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.854666 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.854569 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-installation-pull-secrets\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.854666 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.854550 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c36255e9-35a1-4c63-8154-aaa712e37e96-cert\") pod \"ingress-canary-nmchs\" (UID: \"c36255e9-35a1-4c63-8154-aaa712e37e96\") " pod="openshift-ingress-canary/ingress-canary-nmchs" Jul 07 14:57:08.863984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.863954 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtcb\" (UniqueName: \"kubernetes.io/projected/c36255e9-35a1-4c63-8154-aaa712e37e96-kube-api-access-jjtcb\") pod \"ingress-canary-nmchs\" (UID: \"c36255e9-35a1-4c63-8154-aaa712e37e96\") " pod="openshift-ingress-canary/ingress-canary-nmchs" Jul 07 14:57:08.864090 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.863984 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-bound-sa-token\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.864256 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.864233 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfxpf\" (UniqueName: \"kubernetes.io/projected/a7397841-807c-42d8-b55f-920d4adeb369-kube-api-access-jfxpf\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:08.865883 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.865858 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5kt\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-kube-api-access-8t5kt\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:08.954061 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:08.954026 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nmchs" Jul 07 14:57:09.098105 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.097849 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nmchs"] Jul 07 14:57:09.102540 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:09.102511 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36255e9_35a1_4c63_8154_aaa712e37e96.slice/crio-2c05a6116a0f931aa42617bfb39d5efec34c050f3ae901fe3c73a6e864a057e3 WatchSource:0}: Error finding container 2c05a6116a0f931aa42617bfb39d5efec34c050f3ae901fe3c73a6e864a057e3: Status 404 returned error can't find the container with id 2c05a6116a0f931aa42617bfb39d5efec34c050f3ae901fe3c73a6e864a057e3 Jul 07 14:57:09.306478 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.306443 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:09.306637 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.306453 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:57:09.306637 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.306453 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:09.310393 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.310372 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:57:09.310661 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.310645 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w68zb\"" Jul 07 14:57:09.310739 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.310669 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:57:09.310739 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.310715 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:57:09.310849 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.310647 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:57:09.311939 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.311919 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nm9h4\"" Jul 07 14:57:09.353501 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.353446 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:09.353618 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.353516 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:09.353618 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:09.353606 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:57:09.353728 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:09.353620 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:57:09.353728 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:09.353637 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696d4545c5-pc6zb: secret "image-registry-tls" not found Jul 07 14:57:09.353728 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:09.353664 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls podName:a7397841-807c-42d8-b55f-920d4adeb369 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:10.353646468 +0000 UTC m=+37.561704479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls") pod "dns-default-lq95g" (UID: "a7397841-807c-42d8-b55f-920d4adeb369") : secret "dns-default-metrics-tls" not found Jul 07 14:57:09.353857 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:09.353731 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls podName:02236daa-01c6-4fcf-b497-444d8258a20b nodeName:}" failed. No retries permitted until 2026-07-07 14:57:10.35371109 +0000 UTC m=+37.561769114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls") pod "image-registry-696d4545c5-pc6zb" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b") : secret "image-registry-tls" not found Jul 07 14:57:09.592168 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:09.592136 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nmchs" event={"ID":"c36255e9-35a1-4c63-8154-aaa712e37e96","Type":"ContainerStarted","Data":"2c05a6116a0f931aa42617bfb39d5efec34c050f3ae901fe3c73a6e864a057e3"} Jul 07 14:57:10.363548 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:10.363510 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:10.363757 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:10.363567 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:10.363757 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:10.363700 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:57:10.363864 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:10.363781 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls podName:a7397841-807c-42d8-b55f-920d4adeb369 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:12.363757866 +0000 UTC m=+39.571815891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls") pod "dns-default-lq95g" (UID: "a7397841-807c-42d8-b55f-920d4adeb369") : secret "dns-default-metrics-tls" not found Jul 07 14:57:10.363864 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:10.363703 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:57:10.363864 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:10.363817 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696d4545c5-pc6zb: secret "image-registry-tls" not found Jul 07 14:57:10.364020 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:10.363887 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls podName:02236daa-01c6-4fcf-b497-444d8258a20b nodeName:}" failed. No retries permitted until 2026-07-07 14:57:12.363870068 +0000 UTC m=+39.571928100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls") pod "image-registry-696d4545c5-pc6zb" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b") : secret "image-registry-tls" not found Jul 07 14:57:12.380821 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:12.380777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:12.381298 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:12.380860 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:12.381298 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:12.380994 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:57:12.381298 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:12.381073 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:57:12.381298 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:12.381089 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696d4545c5-pc6zb: secret "image-registry-tls" not found Jul 07 14:57:12.381298 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:12.381098 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls podName:a7397841-807c-42d8-b55f-920d4adeb369 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:16.381074805 +0000 UTC m=+43.589132836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls") pod "dns-default-lq95g" (UID: "a7397841-807c-42d8-b55f-920d4adeb369") : secret "dns-default-metrics-tls" not found Jul 07 14:57:12.381298 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:12.381130 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls podName:02236daa-01c6-4fcf-b497-444d8258a20b nodeName:}" failed. No retries permitted until 2026-07-07 14:57:16.381118411 +0000 UTC m=+43.589176423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls") pod "image-registry-696d4545c5-pc6zb" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b") : secret "image-registry-tls" not found Jul 07 14:57:13.604499 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:13.604469 2566 generic.go:356] "Generic (PLEG): container finished" podID="92432074-635b-4c94-a0b9-363d83ed7127" containerID="d92829ad2e9f75960ba12b832695f9b80a3e1a502c5e5b0c4ded8c681a563eb2" exitCode=0 Jul 07 14:57:13.604962 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:13.604546 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" event={"ID":"92432074-635b-4c94-a0b9-363d83ed7127","Type":"ContainerDied","Data":"d92829ad2e9f75960ba12b832695f9b80a3e1a502c5e5b0c4ded8c681a563eb2"} Jul 07 14:57:13.605955 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:13.605931 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nmchs" event={"ID":"c36255e9-35a1-4c63-8154-aaa712e37e96","Type":"ContainerStarted","Data":"d970e45d8193950f22f2c3b07abf7c1d3c13a7796d984df637ee45026f7cbccc"} Jul 07 14:57:13.652637 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:13.652578 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nmchs" podStartSLOduration=1.83477026 podStartE2EDuration="5.65255526s" totalImagesPullingTime="3.817785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:13.652247499 +0000 UTC m=+40.860305534" watchObservedRunningTime="2026-07-07 14:57:13.65255526 +0000 UTC m=+40.860613293" Jul 07 14:57:14.614060 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:14.614020 2566 generic.go:356] "Generic (PLEG): container finished" podID="92432074-635b-4c94-a0b9-363d83ed7127" containerID="dacbf745311a6d24e9b0aff1cd4bfc449bccaa90d76c90123dd8ee5a28aad6e0" exitCode=0 Jul 07 14:57:14.614495 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:14.614113 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" event={"ID":"92432074-635b-4c94-a0b9-363d83ed7127","Type":"ContainerDied","Data":"dacbf745311a6d24e9b0aff1cd4bfc449bccaa90d76c90123dd8ee5a28aad6e0"} Jul 07 14:57:15.619709 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:15.619670 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" event={"ID":"92432074-635b-4c94-a0b9-363d83ed7127","Type":"ContainerStarted","Data":"62bef9cf7c5701caf085ffb72caffb8fa9a108fc2510280ea9c187ceda490168"} Jul 07 14:57:15.646341 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:15.646295 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mn7mm" podStartSLOduration=9.439450102 podStartE2EDuration="42.646280985s" totalImagesPullingTime="33.206830883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:15.644348083 +0000 UTC m=+42.852406126" watchObservedRunningTime="2026-07-07 14:57:15.646280985 +0000 UTC m=+42.854339016" Jul 07 14:57:16.409959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:16.409918 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:16.410155 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:16.409974 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:16.410155 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:16.410090 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:57:16.410155 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:16.410114 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:57:16.410155 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:16.410127 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696d4545c5-pc6zb: secret "image-registry-tls" not found Jul 07 14:57:16.410302 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:16.410163 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls podName:a7397841-807c-42d8-b55f-920d4adeb369 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:24.410144465 +0000 UTC m=+51.618202484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls") pod "dns-default-lq95g" (UID: "a7397841-807c-42d8-b55f-920d4adeb369") : secret "dns-default-metrics-tls" not found Jul 07 14:57:16.410302 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:16.410178 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls podName:02236daa-01c6-4fcf-b497-444d8258a20b nodeName:}" failed. No retries permitted until 2026-07-07 14:57:24.410171915 +0000 UTC m=+51.618229925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls") pod "image-registry-696d4545c5-pc6zb" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b") : secret "image-registry-tls" not found Jul 07 14:57:24.470361 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:24.470310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:24.470361 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:24.470373 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:24.470846 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:24.470473 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:57:24.470846 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:24.470476 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:57:24.470846 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:24.470556 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls podName:a7397841-807c-42d8-b55f-920d4adeb369 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:40.470540138 +0000 UTC m=+67.678598148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls") pod "dns-default-lq95g" (UID: "a7397841-807c-42d8-b55f-920d4adeb369") : secret "dns-default-metrics-tls" not found Jul 07 14:57:24.470846 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:24.470488 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696d4545c5-pc6zb: secret "image-registry-tls" not found Jul 07 14:57:24.470846 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:24.470629 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls podName:02236daa-01c6-4fcf-b497-444d8258a20b nodeName:}" failed. No retries permitted until 2026-07-07 14:57:40.470616626 +0000 UTC m=+67.678674652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls") pod "image-registry-696d4545c5-pc6zb" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b") : secret "image-registry-tls" not found Jul 07 14:57:26.108503 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.108468 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-kqd56"] Jul 07 14:57:26.125816 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.125792 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.134166 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.134140 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 14:57:26.135289 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.135272 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 14:57:26.135361 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.135312 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 14:57:26.136882 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.136867 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 14:57:26.148363 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.148336 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5qsj6\"" Jul 07 14:57:26.148944 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.148922 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-kqd56"] Jul 07 14:57:26.182737 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.182706 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 14:57:26.183106 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.183082 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/171153e3-a823-4d76-a913-f83499139e27-tmp\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.183178 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.183119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/171153e3-a823-4d76-a913-f83499139e27-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.183178 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.183159 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/171153e3-a823-4d76-a913-f83499139e27-snapshots\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.183250 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.183193 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x5hc\" (UniqueName: \"kubernetes.io/projected/171153e3-a823-4d76-a913-f83499139e27-kube-api-access-9x5hc\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.183285 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.183276 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/171153e3-a823-4d76-a913-f83499139e27-serving-cert\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.183329 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.183316 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/171153e3-a823-4d76-a913-f83499139e27-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.284722 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.284684 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/171153e3-a823-4d76-a913-f83499139e27-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.284879 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.284774 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/171153e3-a823-4d76-a913-f83499139e27-tmp\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.284879 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.284806 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/171153e3-a823-4d76-a913-f83499139e27-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.284879 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.284831 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/171153e3-a823-4d76-a913-f83499139e27-snapshots\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.284879 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.284848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x5hc\" (UniqueName: \"kubernetes.io/projected/171153e3-a823-4d76-a913-f83499139e27-kube-api-access-9x5hc\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.284879 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.284868 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/171153e3-a823-4d76-a913-f83499139e27-serving-cert\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.285475 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.285441 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/171153e3-a823-4d76-a913-f83499139e27-tmp\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.285590 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.285491 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/171153e3-a823-4d76-a913-f83499139e27-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.285590 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.285516 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/171153e3-a823-4d76-a913-f83499139e27-snapshots\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.285720 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.285702 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/171153e3-a823-4d76-a913-f83499139e27-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.288024 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.288004 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/171153e3-a823-4d76-a913-f83499139e27-serving-cert\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.293321 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.293300 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x5hc\" (UniqueName: \"kubernetes.io/projected/171153e3-a823-4d76-a913-f83499139e27-kube-api-access-9x5hc\") pod \"insights-operator-5c847dcfc4-kqd56\" (UID: \"171153e3-a823-4d76-a913-f83499139e27\") " pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.316283 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.316248 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-8lrt5"] Jul 07 14:57:26.338079 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.338034 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-8lrt5"] Jul 07 14:57:26.338230 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.338162 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.341215 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.341180 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 14:57:26.341215 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.341209 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 14:57:26.341399 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.341215 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gz4sb\"" Jul 07 14:57:26.341399 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.341269 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 14:57:26.341399 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.341284 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:57:26.346860 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.346839 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 14:57:26.385915 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.385827 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24927288-0653-428f-98cf-c7c53e70ab43-trusted-ca\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.386151 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.385957 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24927288-0653-428f-98cf-c7c53e70ab43-config\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.386151 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.385982 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24927288-0653-428f-98cf-c7c53e70ab43-serving-cert\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.386151 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.386001 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5snj\" (UniqueName: \"kubernetes.io/projected/24927288-0653-428f-98cf-c7c53e70ab43-kube-api-access-x5snj\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.437188 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.437161 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" Jul 07 14:57:26.487020 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.486978 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24927288-0653-428f-98cf-c7c53e70ab43-config\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.487248 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.487032 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24927288-0653-428f-98cf-c7c53e70ab43-serving-cert\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.487248 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.487092 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5snj\" (UniqueName: \"kubernetes.io/projected/24927288-0653-428f-98cf-c7c53e70ab43-kube-api-access-x5snj\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.487248 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.487188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24927288-0653-428f-98cf-c7c53e70ab43-trusted-ca\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.488202 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.488166 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24927288-0653-428f-98cf-c7c53e70ab43-trusted-ca\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.488478 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.488451 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24927288-0653-428f-98cf-c7c53e70ab43-config\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.490188 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.490169 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24927288-0653-428f-98cf-c7c53e70ab43-serving-cert\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.496882 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.496853 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5snj\" (UniqueName: \"kubernetes.io/projected/24927288-0653-428f-98cf-c7c53e70ab43-kube-api-access-x5snj\") pod \"console-operator-68b757fb67-8lrt5\" (UID: \"24927288-0653-428f-98cf-c7c53e70ab43\") " pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.577952 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.577920 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-kqd56"] Jul 07 14:57:26.581713 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:26.581681 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171153e3_a823_4d76_a913_f83499139e27.slice/crio-002afddc08ae6f8ee822e78146548183e9aa62df5c9b5a233085d681787abb15 WatchSource:0}: Error finding container 002afddc08ae6f8ee822e78146548183e9aa62df5c9b5a233085d681787abb15: Status 404 returned error can't find the container with id 002afddc08ae6f8ee822e78146548183e9aa62df5c9b5a233085d681787abb15 Jul 07 14:57:26.644229 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.644139 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" event={"ID":"171153e3-a823-4d76-a913-f83499139e27","Type":"ContainerStarted","Data":"002afddc08ae6f8ee822e78146548183e9aa62df5c9b5a233085d681787abb15"} Jul 07 14:57:26.647410 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.647392 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:26.801324 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:26.801290 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-8lrt5"] Jul 07 14:57:26.804691 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:26.804664 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24927288_0653_428f_98cf_c7c53e70ab43.slice/crio-9d862e73a0c4a7d1e831fe5fc270a38b8cb985d2d017bab641dc04bd4376c238 WatchSource:0}: Error finding container 9d862e73a0c4a7d1e831fe5fc270a38b8cb985d2d017bab641dc04bd4376c238: Status 404 returned error can't find the container with id 9d862e73a0c4a7d1e831fe5fc270a38b8cb985d2d017bab641dc04bd4376c238 Jul 07 14:57:27.648743 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:27.648696 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" event={"ID":"24927288-0653-428f-98cf-c7c53e70ab43","Type":"ContainerStarted","Data":"9d862e73a0c4a7d1e831fe5fc270a38b8cb985d2d017bab641dc04bd4376c238"} Jul 07 14:57:29.655318 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:29.655279 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" event={"ID":"171153e3-a823-4d76-a913-f83499139e27","Type":"ContainerStarted","Data":"7924f98de40f3462691a7b54f44dfc700bfae2927abb936132946a9eaa2159a6"} Jul 07 14:57:29.656900 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:29.656880 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/0.log" Jul 07 14:57:29.657022 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:29.656915 2566 generic.go:356] "Generic (PLEG): container finished" podID="24927288-0653-428f-98cf-c7c53e70ab43" containerID="55de9188baec6d8bc07d9e8ec6f690f3492ae1e1df3c9cc7f3bcf266c0c7b158" exitCode=255 Jul 07 14:57:29.657022 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:29.656956 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" event={"ID":"24927288-0653-428f-98cf-c7c53e70ab43","Type":"ContainerDied","Data":"55de9188baec6d8bc07d9e8ec6f690f3492ae1e1df3c9cc7f3bcf266c0c7b158"} Jul 07 14:57:29.657210 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:29.657193 2566 scope.go:117] "RemoveContainer" containerID="55de9188baec6d8bc07d9e8ec6f690f3492ae1e1df3c9cc7f3bcf266c0c7b158" Jul 07 14:57:29.679147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:29.679102 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" podStartSLOduration=1.193817273 podStartE2EDuration="3.67908726s" totalImagesPullingTime="2.485269987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:29.678101186 +0000 UTC m=+56.886159219" watchObservedRunningTime="2026-07-07 14:57:29.67908726 +0000 UTC m=+56.887145296" Jul 07 14:57:30.229499 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.229454 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-csrnj"] Jul 07 14:57:30.232146 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.232115 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.235546 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.235516 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c5zlx\"" Jul 07 14:57:30.236326 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.236295 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 14:57:30.238561 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.238512 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 14:57:30.245099 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.245073 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-csrnj"] Jul 07 14:57:30.319836 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.319804 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4479f7df-b08d-45b4-9c23-dfef45423cb1-data-volume\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.319836 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.319844 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.320064 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.319948 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4479f7df-b08d-45b4-9c23-dfef45423cb1-crio-socket\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.320109 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.320071 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.320142 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.320110 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8bj2\" (UniqueName: \"kubernetes.io/projected/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-api-access-b8bj2\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.421108 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.421073 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4479f7df-b08d-45b4-9c23-dfef45423cb1-crio-socket\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.421277 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.421134 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.421277 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.421179 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8bj2\" (UniqueName: \"kubernetes.io/projected/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-api-access-b8bj2\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.421277 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.421200 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4479f7df-b08d-45b4-9c23-dfef45423cb1-data-volume\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.421277 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.421199 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4479f7df-b08d-45b4-9c23-dfef45423cb1-crio-socket\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.421277 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.421234 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.421431 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:30.421281 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:57:30.421431 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:30.421355 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:30.921336893 +0000 UTC m=+58.129394933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : configmap "kube-rbac-proxy" not found Jul 07 14:57:30.421431 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:30.421368 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:57:30.421431 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:30.421415 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:30.921403926 +0000 UTC m=+58.129461936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : secret "insights-runtime-extractor-tls" not found Jul 07 14:57:30.421558 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.421506 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4479f7df-b08d-45b4-9c23-dfef45423cb1-data-volume\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.432800 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.432770 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8bj2\" (UniqueName: \"kubernetes.io/projected/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-api-access-b8bj2\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.661420 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.661389 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 14:57:30.661878 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.661818 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/0.log" Jul 07 14:57:30.661878 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.661863 2566 generic.go:356] "Generic (PLEG): container finished" podID="24927288-0653-428f-98cf-c7c53e70ab43" containerID="b7e8264bb00a970242ecdbaf7d7deadffd8abc75f5a1a5b66f6221db6e6eab2c" exitCode=255 Jul 07 14:57:30.661977 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.661950 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" event={"ID":"24927288-0653-428f-98cf-c7c53e70ab43","Type":"ContainerDied","Data":"b7e8264bb00a970242ecdbaf7d7deadffd8abc75f5a1a5b66f6221db6e6eab2c"} Jul 07 14:57:30.662117 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.662004 2566 scope.go:117] "RemoveContainer" containerID="55de9188baec6d8bc07d9e8ec6f690f3492ae1e1df3c9cc7f3bcf266c0c7b158" Jul 07 14:57:30.662308 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.662292 2566 scope.go:117] "RemoveContainer" containerID="b7e8264bb00a970242ecdbaf7d7deadffd8abc75f5a1a5b66f6221db6e6eab2c" Jul 07 14:57:30.662492 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:30.662472 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-8lrt5_openshift-console-operator(24927288-0653-428f-98cf-c7c53e70ab43)\"" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" podUID="24927288-0653-428f-98cf-c7c53e70ab43" Jul 07 14:57:30.925494 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.925397 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.925494 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:30.925457 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:30.925708 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:30.925525 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:57:30.925708 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:30.925569 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:57:30.925708 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:30.925594 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:31.925578075 +0000 UTC m=+59.133636085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : configmap "kube-rbac-proxy" not found Jul 07 14:57:30.925708 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:30.925617 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:31.925606285 +0000 UTC m=+59.133664296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : secret "insights-runtime-extractor-tls" not found Jul 07 14:57:31.666317 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:31.666288 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 14:57:31.666683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:31.666632 2566 scope.go:117] "RemoveContainer" containerID="b7e8264bb00a970242ecdbaf7d7deadffd8abc75f5a1a5b66f6221db6e6eab2c" Jul 07 14:57:31.666807 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:31.666783 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-8lrt5_openshift-console-operator(24927288-0653-428f-98cf-c7c53e70ab43)\"" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" podUID="24927288-0653-428f-98cf-c7c53e70ab43" Jul 07 14:57:31.935628 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:31.935538 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:31.935781 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:31.935657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:31.935781 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:31.935708 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:57:31.935861 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:31.935780 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:57:31.935861 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:31.935786 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:33.935768478 +0000 UTC m=+61.143826491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : secret "insights-runtime-extractor-tls" not found Jul 07 14:57:31.935861 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:31.935849 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:33.935829989 +0000 UTC m=+61.143888013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : configmap "kube-rbac-proxy" not found Jul 07 14:57:32.823534 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:32.823503 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f74m_ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac/dns-node-resolver/0.log" Jul 07 14:57:33.831405 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:33.831369 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dlvgv_627d2b1b-5d4b-416b-ad66-687277f43add/node-ca/0.log" Jul 07 14:57:33.951232 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:33.951194 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:33.951397 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:33.951258 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:33.951397 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:33.951336 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:57:33.951397 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:33.951365 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:57:33.951516 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:33.951424 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:37.951409595 +0000 UTC m=+65.159467605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : secret "insights-runtime-extractor-tls" not found Jul 07 14:57:33.951516 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:33.951437 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:37.95143108 +0000 UTC m=+65.159489090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : configmap "kube-rbac-proxy" not found Jul 07 14:57:34.587913 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:34.587881 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw2hn" Jul 07 14:57:34.826851 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:34.826822 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nmchs_c36255e9-35a1-4c63-8154-aaa712e37e96/serve-healthcheck-canary/0.log" Jul 07 14:57:36.647757 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:36.647715 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:36.648261 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:36.647809 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:36.648337 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:36.648258 2566 scope.go:117] "RemoveContainer" containerID="b7e8264bb00a970242ecdbaf7d7deadffd8abc75f5a1a5b66f6221db6e6eab2c" Jul 07 14:57:36.648460 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:36.648436 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-8lrt5_openshift-console-operator(24927288-0653-428f-98cf-c7c53e70ab43)\"" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" podUID="24927288-0653-428f-98cf-c7c53e70ab43" Jul 07 14:57:37.984219 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:37.984182 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:37.984669 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:37.984259 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:37.984669 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:37.984304 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:57:37.984669 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:37.984364 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:45.984349991 +0000 UTC m=+73.192408006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : configmap "kube-rbac-proxy" not found Jul 07 14:57:37.984669 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:37.984442 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:57:37.984669 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:37.984518 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls podName:4479f7df-b08d-45b4-9c23-dfef45423cb1 nodeName:}" failed. No retries permitted until 2026-07-07 14:57:45.984499265 +0000 UTC m=+73.192557296 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-csrnj" (UID: "4479f7df-b08d-45b4-9c23-dfef45423cb1") : secret "insights-runtime-extractor-tls" not found Jul 07 14:57:38.992811 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:38.992777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:57:38.995641 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:38.995622 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:57:39.003117 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:39.003096 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 14:57:39.003254 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:57:39.003152 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs podName:f115fbf4-80c0-4991-b663-61ce83d6e968 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:43.003138513 +0000 UTC m=+130.211196524 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs") pod "network-metrics-daemon-jw459" (UID: "f115fbf4-80c0-4991-b663-61ce83d6e968") : secret "metrics-daemon-secret" not found Jul 07 14:57:39.093235 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.093202 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:39.093427 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.093245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:39.096592 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.096570 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:57:39.096652 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.096570 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:57:39.106421 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.106394 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c09a0b2-4116-49e0-9727-aa0de72a0b22-original-pull-secret\") pod \"global-pull-secret-syncer-kwswc\" (UID: \"9c09a0b2-4116-49e0-9727-aa0de72a0b22\") " pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:39.106637 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.106623 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:57:39.116603 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.116578 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dwqn\" (UniqueName: \"kubernetes.io/projected/0c6588bc-27c9-491e-aac7-b6b1597fdfe0-kube-api-access-7dwqn\") pod \"network-check-target-mvn5b\" (UID: \"0c6588bc-27c9-491e-aac7-b6b1597fdfe0\") " pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:39.322617 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.322588 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w68zb\"" Jul 07 14:57:39.329646 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.329613 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:39.331840 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.331821 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kwswc" Jul 07 14:57:39.475967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.475937 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kwswc"] Jul 07 14:57:39.479406 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:39.479382 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c09a0b2_4116_49e0_9727_aa0de72a0b22.slice/crio-01e35a31343f49a6ddf45b7437012b2eee34267534227a5b0b9244867ceed4eb WatchSource:0}: Error finding container 01e35a31343f49a6ddf45b7437012b2eee34267534227a5b0b9244867ceed4eb: Status 404 returned error can't find the container with id 01e35a31343f49a6ddf45b7437012b2eee34267534227a5b0b9244867ceed4eb Jul 07 14:57:39.497432 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.497411 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mvn5b"] Jul 07 14:57:39.500344 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:39.500322 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6588bc_27c9_491e_aac7_b6b1597fdfe0.slice/crio-f279e225620779cef9979bda8d0ddc83dd871fe803324de7672a4bc885b47a99 WatchSource:0}: Error finding container f279e225620779cef9979bda8d0ddc83dd871fe803324de7672a4bc885b47a99: Status 404 returned error can't find the container with id f279e225620779cef9979bda8d0ddc83dd871fe803324de7672a4bc885b47a99 Jul 07 14:57:39.687840 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.687763 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kwswc" event={"ID":"9c09a0b2-4116-49e0-9727-aa0de72a0b22","Type":"ContainerStarted","Data":"01e35a31343f49a6ddf45b7437012b2eee34267534227a5b0b9244867ceed4eb"} Jul 07 14:57:39.688717 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:39.688697 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mvn5b" event={"ID":"0c6588bc-27c9-491e-aac7-b6b1597fdfe0","Type":"ContainerStarted","Data":"f279e225620779cef9979bda8d0ddc83dd871fe803324de7672a4bc885b47a99"} Jul 07 14:57:40.504674 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.504641 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:40.505200 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.504706 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:40.507984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.507927 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"image-registry-696d4545c5-pc6zb\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:40.508136 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.508103 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7397841-807c-42d8-b55f-920d4adeb369-metrics-tls\") pod \"dns-default-lq95g\" (UID: \"a7397841-807c-42d8-b55f-920d4adeb369\") " pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:40.721089 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.721059 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dszn9\"" Jul 07 14:57:40.728557 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.728528 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:40.747088 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.747061 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9hq6c\"" Jul 07 14:57:40.754658 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.754620 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:40.913014 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.912807 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-696d4545c5-pc6zb"] Jul 07 14:57:40.916954 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:40.916926 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02236daa_01c6_4fcf_b497_444d8258a20b.slice/crio-4d2c2527b09f80f6a96d7b7025fd55ec09f6bbd4e7c455e1a7ee4f4fc0da39bd WatchSource:0}: Error finding container 4d2c2527b09f80f6a96d7b7025fd55ec09f6bbd4e7c455e1a7ee4f4fc0da39bd: Status 404 returned error can't find the container with id 4d2c2527b09f80f6a96d7b7025fd55ec09f6bbd4e7c455e1a7ee4f4fc0da39bd Jul 07 14:57:40.930388 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:40.930362 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lq95g"] Jul 07 14:57:40.934378 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:40.934341 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7397841_807c_42d8_b55f_920d4adeb369.slice/crio-df7869db051dbe60ce068113618ef2e8565d6b1b9493b2ad44cc4fb6e5e1bb35 WatchSource:0}: Error finding container df7869db051dbe60ce068113618ef2e8565d6b1b9493b2ad44cc4fb6e5e1bb35: Status 404 returned error can't find the container with id df7869db051dbe60ce068113618ef2e8565d6b1b9493b2ad44cc4fb6e5e1bb35 Jul 07 14:57:41.697436 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:41.697321 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" event={"ID":"02236daa-01c6-4fcf-b497-444d8258a20b","Type":"ContainerStarted","Data":"de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b"} Jul 07 14:57:41.697436 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:41.697382 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" event={"ID":"02236daa-01c6-4fcf-b497-444d8258a20b","Type":"ContainerStarted","Data":"4d2c2527b09f80f6a96d7b7025fd55ec09f6bbd4e7c455e1a7ee4f4fc0da39bd"} Jul 07 14:57:41.697436 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:41.697406 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:57:41.699323 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:41.699291 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lq95g" event={"ID":"a7397841-807c-42d8-b55f-920d4adeb369","Type":"ContainerStarted","Data":"df7869db051dbe60ce068113618ef2e8565d6b1b9493b2ad44cc4fb6e5e1bb35"} Jul 07 14:57:41.725612 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:41.725276 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" podStartSLOduration=68.725256158 podStartE2EDuration="1m8.725256158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:41.72317466 +0000 UTC m=+68.931232705" watchObservedRunningTime="2026-07-07 14:57:41.725256158 +0000 UTC m=+68.933314191" Jul 07 14:57:45.714090 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:45.714035 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mvn5b" event={"ID":"0c6588bc-27c9-491e-aac7-b6b1597fdfe0","Type":"ContainerStarted","Data":"fc91f44cdb6f11aaa2a7879031e94a64069f9f9ff520aa100d2b858f1364a1f8"} Jul 07 14:57:45.714547 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:45.714139 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:57:45.715840 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:45.715813 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lq95g" event={"ID":"a7397841-807c-42d8-b55f-920d4adeb369","Type":"ContainerStarted","Data":"019055bdc532fdedec2996a833cc6880f1d83012a5309c6260627124664ddd45"} Jul 07 14:57:45.715991 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:45.715860 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lq95g" event={"ID":"a7397841-807c-42d8-b55f-920d4adeb369","Type":"ContainerStarted","Data":"c860925b2e0a058368e8e2d97f11c60f9ffccfae8806f8cd776591a7313881e1"} Jul 07 14:57:45.715991 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:45.715888 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:45.717187 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:45.717154 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kwswc" event={"ID":"9c09a0b2-4116-49e0-9727-aa0de72a0b22","Type":"ContainerStarted","Data":"40d2ff2887af472e47b7eadc32e9d86fdd731a819bb5e304682267c6343cfa84"} Jul 07 14:57:45.749079 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:45.749019 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mvn5b" podStartSLOduration=67.573334145 podStartE2EDuration="1m12.749005775s" totalImagesPullingTime="5.17567163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:45.733329683 +0000 UTC m=+72.941387716" watchObservedRunningTime="2026-07-07 14:57:45.749005775 +0000 UTC m=+72.957063785" Jul 07 14:57:45.749321 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:45.749301 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kwswc" podStartSLOduration=67.540311239 podStartE2EDuration="1m12.749294254s" totalImagesPullingTime="5.208983015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:45.74833212 +0000 UTC m=+72.956390165" watchObservedRunningTime="2026-07-07 14:57:45.749294254 +0000 UTC m=+72.957352286" Jul 07 14:57:45.768841 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:45.768802 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lq95g" podStartSLOduration=34.025358533 podStartE2EDuration="37.768789667s" totalImagesPullingTime="3.743431134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:45.768653376 +0000 UTC m=+72.976711408" watchObservedRunningTime="2026-07-07 14:57:45.768789667 +0000 UTC m=+72.976847690" Jul 07 14:57:46.053647 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:46.053613 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:46.053835 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:46.053661 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:46.054218 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:46.054193 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4479f7df-b08d-45b4-9c23-dfef45423cb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:46.055877 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:46.055860 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4479f7df-b08d-45b4-9c23-dfef45423cb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-csrnj\" (UID: \"4479f7df-b08d-45b4-9c23-dfef45423cb1\") " pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:46.147307 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:46.147287 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c5zlx\"" Jul 07 14:57:46.154673 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:46.154655 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-csrnj" Jul 07 14:57:46.300921 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:46.300785 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-csrnj"] Jul 07 14:57:46.303477 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:46.303450 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4479f7df_b08d_45b4_9c23_dfef45423cb1.slice/crio-d84bd7c94b64b02c669143e6c5f9610258801807f742cbf577197c6cd9302b8c WatchSource:0}: Error finding container d84bd7c94b64b02c669143e6c5f9610258801807f742cbf577197c6cd9302b8c: Status 404 returned error can't find the container with id d84bd7c94b64b02c669143e6c5f9610258801807f742cbf577197c6cd9302b8c Jul 07 14:57:46.721984 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:46.721895 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-csrnj" event={"ID":"4479f7df-b08d-45b4-9c23-dfef45423cb1","Type":"ContainerStarted","Data":"d84bd7c94b64b02c669143e6c5f9610258801807f742cbf577197c6cd9302b8c"} Jul 07 14:57:49.306323 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:49.306286 2566 scope.go:117] "RemoveContainer" containerID="b7e8264bb00a970242ecdbaf7d7deadffd8abc75f5a1a5b66f6221db6e6eab2c" Jul 07 14:57:50.738331 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:50.738295 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-csrnj" event={"ID":"4479f7df-b08d-45b4-9c23-dfef45423cb1","Type":"ContainerStarted","Data":"a4cf374a12e3012b89bd0269a772809b44ecb08a3961750f33b3d996e589334b"} Jul 07 14:57:50.740144 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:50.740123 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 14:57:50.740271 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:50.740169 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" event={"ID":"24927288-0653-428f-98cf-c7c53e70ab43","Type":"ContainerStarted","Data":"086839c9560d08c32f94eee5ac8c69d85303fa9505681df1766ab2f3ebe2b42b"} Jul 07 14:57:50.740411 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:50.740393 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:50.744606 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:50.744588 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" Jul 07 14:57:50.784094 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:50.784040 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-8lrt5" podStartSLOduration=22.52061159 podStartE2EDuration="24.784027499s" totalImagesPullingTime="2.263415909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:50.783848348 +0000 UTC m=+77.991906380" watchObservedRunningTime="2026-07-07 14:57:50.784027499 +0000 UTC m=+77.992085530" Jul 07 14:57:51.746328 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:51.746285 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-csrnj" event={"ID":"4479f7df-b08d-45b4-9c23-dfef45423cb1","Type":"ContainerStarted","Data":"0625292e8ccd4203638d3f69629ec3263ef6300f03db9f6e3d36eefe411046ad"} Jul 07 14:57:53.756412 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:53.756377 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-csrnj" event={"ID":"4479f7df-b08d-45b4-9c23-dfef45423cb1","Type":"ContainerStarted","Data":"c2d9936e4d241c07c40c26711bee71b97f9452060365144768d25e9ac786cba0"} Jul 07 14:57:53.785118 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:53.785074 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-csrnj" podStartSLOduration=16.861737804 podStartE2EDuration="23.78503796s" totalImagesPullingTime="6.923300156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:57:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:53.784439724 +0000 UTC m=+80.992497755" watchObservedRunningTime="2026-07-07 14:57:53.78503796 +0000 UTC m=+80.993095992" Jul 07 14:57:54.380240 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.380207 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8"] Jul 07 14:57:54.382676 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.382662 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" Jul 07 14:57:54.388886 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.388868 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 14:57:54.389338 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.389322 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 14:57:54.389449 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.389432 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vll8k\"" Jul 07 14:57:54.390275 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.390260 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 14:57:54.416220 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.416195 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 14:57:54.435944 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.435919 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8"] Jul 07 14:57:54.441523 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.441503 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-9nghg"] Jul 07 14:57:54.443941 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.443924 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9nghg" Jul 07 14:57:54.449811 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.449791 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 14:57:54.452147 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.452133 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zbbth\"" Jul 07 14:57:54.458187 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.458173 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 14:57:54.466341 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.466318 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9nghg"] Jul 07 14:57:54.528784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.528757 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6wj8\" (UniqueName: \"kubernetes.io/projected/4a73abd4-1b52-492a-88f9-a251d4883bfe-kube-api-access-c6wj8\") pod \"managed-serviceaccount-addon-agent-58bd49567d-sfzs8\" (UID: \"4a73abd4-1b52-492a-88f9-a251d4883bfe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" Jul 07 14:57:54.528891 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.528799 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a73abd4-1b52-492a-88f9-a251d4883bfe-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58bd49567d-sfzs8\" (UID: \"4a73abd4-1b52-492a-88f9-a251d4883bfe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" Jul 07 14:57:54.528891 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.528841 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bczv\" (UniqueName: \"kubernetes.io/projected/68f1313e-1e49-47e6-89af-a00a13e35f38-kube-api-access-7bczv\") pod \"downloads-685bfb5f64-9nghg\" (UID: \"68f1313e-1e49-47e6-89af-a00a13e35f38\") " pod="openshift-console/downloads-685bfb5f64-9nghg" Jul 07 14:57:54.629778 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.629745 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6wj8\" (UniqueName: \"kubernetes.io/projected/4a73abd4-1b52-492a-88f9-a251d4883bfe-kube-api-access-c6wj8\") pod \"managed-serviceaccount-addon-agent-58bd49567d-sfzs8\" (UID: \"4a73abd4-1b52-492a-88f9-a251d4883bfe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" Jul 07 14:57:54.629778 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.629779 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a73abd4-1b52-492a-88f9-a251d4883bfe-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58bd49567d-sfzs8\" (UID: \"4a73abd4-1b52-492a-88f9-a251d4883bfe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" Jul 07 14:57:54.629967 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.629843 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bczv\" (UniqueName: \"kubernetes.io/projected/68f1313e-1e49-47e6-89af-a00a13e35f38-kube-api-access-7bczv\") pod \"downloads-685bfb5f64-9nghg\" (UID: \"68f1313e-1e49-47e6-89af-a00a13e35f38\") " pod="openshift-console/downloads-685bfb5f64-9nghg" Jul 07 14:57:54.632313 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.632258 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a73abd4-1b52-492a-88f9-a251d4883bfe-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58bd49567d-sfzs8\" (UID: \"4a73abd4-1b52-492a-88f9-a251d4883bfe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" Jul 07 14:57:54.640217 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.640192 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bczv\" (UniqueName: \"kubernetes.io/projected/68f1313e-1e49-47e6-89af-a00a13e35f38-kube-api-access-7bczv\") pod \"downloads-685bfb5f64-9nghg\" (UID: \"68f1313e-1e49-47e6-89af-a00a13e35f38\") " pod="openshift-console/downloads-685bfb5f64-9nghg" Jul 07 14:57:54.640300 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.640222 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6wj8\" (UniqueName: \"kubernetes.io/projected/4a73abd4-1b52-492a-88f9-a251d4883bfe-kube-api-access-c6wj8\") pod \"managed-serviceaccount-addon-agent-58bd49567d-sfzs8\" (UID: \"4a73abd4-1b52-492a-88f9-a251d4883bfe\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" Jul 07 14:57:54.699991 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.699951 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" Jul 07 14:57:54.752328 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.752299 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9nghg" Jul 07 14:57:54.841355 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:54.841323 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a73abd4_1b52_492a_88f9_a251d4883bfe.slice/crio-b983725eb382e70cc430e1a621dea8131dada8cfadda14667841aa866e6a52bb WatchSource:0}: Error finding container b983725eb382e70cc430e1a621dea8131dada8cfadda14667841aa866e6a52bb: Status 404 returned error can't find the container with id b983725eb382e70cc430e1a621dea8131dada8cfadda14667841aa866e6a52bb Jul 07 14:57:54.846008 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.845979 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8"] Jul 07 14:57:54.901450 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:54.901432 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9nghg"] Jul 07 14:57:54.903403 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:57:54.903366 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f1313e_1e49_47e6_89af_a00a13e35f38.slice/crio-650201a7947db977da601d9bcacc73a56ed0a0371af9fdd91181ba05007018e7 WatchSource:0}: Error finding container 650201a7947db977da601d9bcacc73a56ed0a0371af9fdd91181ba05007018e7: Status 404 returned error can't find the container with id 650201a7947db977da601d9bcacc73a56ed0a0371af9fdd91181ba05007018e7 Jul 07 14:57:55.724563 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:55.724296 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lq95g" Jul 07 14:57:55.782330 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:55.782287 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" event={"ID":"4a73abd4-1b52-492a-88f9-a251d4883bfe","Type":"ContainerStarted","Data":"b983725eb382e70cc430e1a621dea8131dada8cfadda14667841aa866e6a52bb"} Jul 07 14:57:55.783965 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:55.783816 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9nghg" event={"ID":"68f1313e-1e49-47e6-89af-a00a13e35f38","Type":"ContainerStarted","Data":"650201a7947db977da601d9bcacc73a56ed0a0371af9fdd91181ba05007018e7"} Jul 07 14:57:57.794534 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:57:57.794497 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" event={"ID":"4a73abd4-1b52-492a-88f9-a251d4883bfe","Type":"ContainerStarted","Data":"a6e4011d6ab58872527f08447960d2080cb690ae53cd29dd7faec8ab51174064"} Jul 07 14:58:00.734545 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:00.734503 2566 patch_prober.go:28] interesting pod/image-registry-696d4545c5-pc6zb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:58:00.734970 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:00.734565 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" podUID="02236daa-01c6-4fcf-b497-444d8258a20b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:58:01.420625 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.420575 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58bd49567d-sfzs8" podStartSLOduration=4.599936439 podStartE2EDuration="7.420553843s" totalImagesPullingTime="2.820617404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:57.818088112 +0000 UTC m=+85.026146144" watchObservedRunningTime="2026-07-07 14:58:01.420553843 +0000 UTC m=+88.628611910" Jul 07 14:58:01.421356 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.421338 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb"] Jul 07 14:58:01.423982 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.423967 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" Jul 07 14:58:01.428018 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.427997 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jbqjf\"" Jul 07 14:58:01.428136 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.427998 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 07 14:58:01.436124 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.436102 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb"] Jul 07 14:58:01.589466 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.589433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43c78cfa-7ffd-444f-ad38-29847a38ba9a-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6pfxb\" (UID: \"43c78cfa-7ffd-444f-ad38-29847a38ba9a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" Jul 07 14:58:01.671098 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.671017 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-697c8bdf69-khv5j"] Jul 07 14:58:01.673545 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.673521 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.678930 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.678898 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 14:58:01.679074 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.678929 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 14:58:01.679074 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.678898 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dp7hd\"" Jul 07 14:58:01.679465 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.679445 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 14:58:01.679591 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.679448 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 14:58:01.680350 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.680332 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 14:58:01.687829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.687808 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-697c8bdf69-khv5j"] Jul 07 14:58:01.691037 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.690648 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43c78cfa-7ffd-444f-ad38-29847a38ba9a-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6pfxb\" (UID: \"43c78cfa-7ffd-444f-ad38-29847a38ba9a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" Jul 07 14:58:01.693505 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.693473 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/43c78cfa-7ffd-444f-ad38-29847a38ba9a-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6pfxb\" (UID: \"43c78cfa-7ffd-444f-ad38-29847a38ba9a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" Jul 07 14:58:01.732694 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.732660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" Jul 07 14:58:01.792150 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.791770 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-serving-cert\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.792150 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.791862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-oauth-config\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.792150 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.791895 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzrg4\" (UniqueName: \"kubernetes.io/projected/ee76827a-c9a7-4851-9e41-a268db8c3043-kube-api-access-hzrg4\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.792150 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.791951 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-console-config\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.792150 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.791991 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-service-ca\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.792150 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.792015 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-oauth-serving-cert\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.877990 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.877954 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb"] Jul 07 14:58:01.881112 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:58:01.881082 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c78cfa_7ffd_444f_ad38_29847a38ba9a.slice/crio-fd59b30e43604dc6334ca6e0b34af9360c657ef3ce03028ea25db6ec8ea1b50a WatchSource:0}: Error finding container fd59b30e43604dc6334ca6e0b34af9360c657ef3ce03028ea25db6ec8ea1b50a: Status 404 returned error can't find the container with id fd59b30e43604dc6334ca6e0b34af9360c657ef3ce03028ea25db6ec8ea1b50a Jul 07 14:58:01.892785 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.892763 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-service-ca\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.892872 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.892806 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-oauth-serving-cert\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.892872 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.892843 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-serving-cert\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.892954 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.892892 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-oauth-config\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.892954 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.892920 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzrg4\" (UniqueName: \"kubernetes.io/projected/ee76827a-c9a7-4851-9e41-a268db8c3043-kube-api-access-hzrg4\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.893101 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.893072 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-console-config\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.893649 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.893594 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-oauth-serving-cert\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.893649 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.893594 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-service-ca\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.893801 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.893767 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-console-config\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.895677 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.895657 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-oauth-config\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.895926 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.895905 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-serving-cert\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.903305 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.903285 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzrg4\" (UniqueName: \"kubernetes.io/projected/ee76827a-c9a7-4851-9e41-a268db8c3043-kube-api-access-hzrg4\") pod \"console-697c8bdf69-khv5j\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:01.984624 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:01.984549 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:02.165163 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:02.165128 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-697c8bdf69-khv5j"] Jul 07 14:58:02.168347 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:58:02.168309 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee76827a_c9a7_4851_9e41_a268db8c3043.slice/crio-4d798c684739c41385a3bd098154e091158f3114ba8cc3befc70efd6213af470 WatchSource:0}: Error finding container 4d798c684739c41385a3bd098154e091158f3114ba8cc3befc70efd6213af470: Status 404 returned error can't find the container with id 4d798c684739c41385a3bd098154e091158f3114ba8cc3befc70efd6213af470 Jul 07 14:58:02.708542 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:02.708514 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:58:02.819750 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:02.819667 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" event={"ID":"43c78cfa-7ffd-444f-ad38-29847a38ba9a","Type":"ContainerStarted","Data":"fd59b30e43604dc6334ca6e0b34af9360c657ef3ce03028ea25db6ec8ea1b50a"} Jul 07 14:58:02.821600 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:02.821569 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697c8bdf69-khv5j" event={"ID":"ee76827a-c9a7-4851-9e41-a268db8c3043","Type":"ContainerStarted","Data":"4d798c684739c41385a3bd098154e091158f3114ba8cc3befc70efd6213af470"} Jul 07 14:58:03.827968 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:03.827913 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" event={"ID":"43c78cfa-7ffd-444f-ad38-29847a38ba9a","Type":"ContainerStarted","Data":"07dffbe79fae27f03b7e07138e301fd641ed086aad705b8da78fe6553a90761b"} Jul 07 14:58:03.828447 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:03.828139 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" Jul 07 14:58:03.834011 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:03.833987 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" Jul 07 14:58:03.863241 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:03.863173 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6pfxb" podStartSLOduration=1.7363207840000001 podStartE2EDuration="2.863150387s" totalImagesPullingTime="1.126829603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:03.845387865 +0000 UTC m=+91.053445897" watchObservedRunningTime="2026-07-07 14:58:03.863150387 +0000 UTC m=+91.071208415" Jul 07 14:58:04.528529 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.528491 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v"] Jul 07 14:58:04.531233 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.531207 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.536989 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.536969 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 14:58:04.538966 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.538930 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fdg9g\"" Jul 07 14:58:04.538966 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.538950 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 14:58:04.539183 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.538980 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 14:58:04.539183 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.538930 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 14:58:04.539183 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.538942 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 14:58:04.552224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.552156 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v"] Jul 07 14:58:04.612799 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.612758 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbrnp\" (UniqueName: \"kubernetes.io/projected/5734779e-8983-4e75-9a6e-4787d908cb85-kube-api-access-sbrnp\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.612994 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.612807 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5734779e-8983-4e75-9a6e-4787d908cb85-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.612994 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.612936 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5734779e-8983-4e75-9a6e-4787d908cb85-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.612994 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.612983 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5734779e-8983-4e75-9a6e-4787d908cb85-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.714330 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.714283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbrnp\" (UniqueName: \"kubernetes.io/projected/5734779e-8983-4e75-9a6e-4787d908cb85-kube-api-access-sbrnp\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.714330 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.714332 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5734779e-8983-4e75-9a6e-4787d908cb85-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.714685 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.714427 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5734779e-8983-4e75-9a6e-4787d908cb85-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.714685 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.714466 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5734779e-8983-4e75-9a6e-4787d908cb85-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.714685 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:58:04.714568 2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 07 14:58:04.714685 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:58:04.714625 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5734779e-8983-4e75-9a6e-4787d908cb85-prometheus-operator-tls podName:5734779e-8983-4e75-9a6e-4787d908cb85 nodeName:}" failed. No retries permitted until 2026-07-07 14:58:05.214605277 +0000 UTC m=+92.422663310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5734779e-8983-4e75-9a6e-4787d908cb85-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-v6x9v" (UID: "5734779e-8983-4e75-9a6e-4787d908cb85") : secret "prometheus-operator-tls" not found Jul 07 14:58:04.715566 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.715539 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5734779e-8983-4e75-9a6e-4787d908cb85-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.717575 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.717547 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5734779e-8983-4e75-9a6e-4787d908cb85-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:04.724325 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:04.724302 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbrnp\" (UniqueName: \"kubernetes.io/projected/5734779e-8983-4e75-9a6e-4787d908cb85-kube-api-access-sbrnp\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:05.218350 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:05.218204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5734779e-8983-4e75-9a6e-4787d908cb85-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:05.221381 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:05.221323 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5734779e-8983-4e75-9a6e-4787d908cb85-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-v6x9v\" (UID: \"5734779e-8983-4e75-9a6e-4787d908cb85\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:05.443571 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:05.443534 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" Jul 07 14:58:10.650311 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.650275 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5968b76755-zvq54"] Jul 07 14:58:10.800140 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.800099 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5968b76755-zvq54"] Jul 07 14:58:10.800325 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.800155 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.810729 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.810530 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 14:58:10.872529 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.872483 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-oauth-config\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.872529 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.872536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-trusted-ca-bundle\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.872777 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.872605 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-config\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.872777 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.872657 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-service-ca\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.872777 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.872703 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-serving-cert\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.872777 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.872734 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-oauth-serving-cert\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.872954 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.872824 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxs7k\" (UniqueName: \"kubernetes.io/projected/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-kube-api-access-lxs7k\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.973905 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.973815 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-serving-cert\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.973905 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.973860 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-oauth-serving-cert\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.973905 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.973890 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxs7k\" (UniqueName: \"kubernetes.io/projected/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-kube-api-access-lxs7k\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.974272 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.973949 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-oauth-config\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.974272 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.974035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-trusted-ca-bundle\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.974272 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.974112 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-config\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.974272 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.974143 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-service-ca\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.975038 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.975012 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-service-ca\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.975184 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.975095 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-config\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.975184 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.975162 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-oauth-serving-cert\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.975324 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.975256 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-trusted-ca-bundle\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.976834 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.976811 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-oauth-config\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.976938 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.976863 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-serving-cert\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:10.983656 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:10.983631 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxs7k\" (UniqueName: \"kubernetes.io/projected/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-kube-api-access-lxs7k\") pod \"console-5968b76755-zvq54\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:11.111786 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:11.111744 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:13.119948 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.119877 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5968b76755-zvq54"] Jul 07 14:58:13.124840 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:58:13.124812 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbff8bbe_6289_484e_a99c_38c0af9ff6a9.slice/crio-66bd7f22a9d19ad0a7f44ed2d9542d7e5a309a886a91c6cd3b451f9043a32a78 WatchSource:0}: Error finding container 66bd7f22a9d19ad0a7f44ed2d9542d7e5a309a886a91c6cd3b451f9043a32a78: Status 404 returned error can't find the container with id 66bd7f22a9d19ad0a7f44ed2d9542d7e5a309a886a91c6cd3b451f9043a32a78 Jul 07 14:58:13.161456 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.161426 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v"] Jul 07 14:58:13.166187 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:58:13.166160 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5734779e_8983_4e75_9a6e_4787d908cb85.slice/crio-cde7f9c95f28487f9fd74f654df7c55e1af139047e8de0aab833ea2de5b0b165 WatchSource:0}: Error finding container cde7f9c95f28487f9fd74f654df7c55e1af139047e8de0aab833ea2de5b0b165: Status 404 returned error can't find the container with id cde7f9c95f28487f9fd74f654df7c55e1af139047e8de0aab833ea2de5b0b165 Jul 07 14:58:13.868915 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.868874 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697c8bdf69-khv5j" event={"ID":"ee76827a-c9a7-4851-9e41-a268db8c3043","Type":"ContainerStarted","Data":"efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75"} Jul 07 14:58:13.870794 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.870758 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" event={"ID":"5734779e-8983-4e75-9a6e-4787d908cb85","Type":"ContainerStarted","Data":"cde7f9c95f28487f9fd74f654df7c55e1af139047e8de0aab833ea2de5b0b165"} Jul 07 14:58:13.874753 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.874720 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5968b76755-zvq54" event={"ID":"fbff8bbe-6289-484e-a99c-38c0af9ff6a9","Type":"ContainerStarted","Data":"16369ad04d1c570fb40ebaa867e94e40153774919540c33baa6d1387c32c381f"} Jul 07 14:58:13.874906 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.874771 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5968b76755-zvq54" event={"ID":"fbff8bbe-6289-484e-a99c-38c0af9ff6a9","Type":"ContainerStarted","Data":"66bd7f22a9d19ad0a7f44ed2d9542d7e5a309a886a91c6cd3b451f9043a32a78"} Jul 07 14:58:13.878237 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.877398 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9nghg" event={"ID":"68f1313e-1e49-47e6-89af-a00a13e35f38","Type":"ContainerStarted","Data":"d6a092d68cbd64a1e97b3910a06c763c2527aa87296c778c4b126cbb6f125aef"} Jul 07 14:58:13.878237 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.878180 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-9nghg" Jul 07 14:58:13.899023 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.894088 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-9nghg" Jul 07 14:58:13.901212 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.899902 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-697c8bdf69-khv5j" podStartSLOduration=2.102036884 podStartE2EDuration="12.89988036s" totalImagesPullingTime="10.797843476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:13.895383045 +0000 UTC m=+101.103441078" watchObservedRunningTime="2026-07-07 14:58:13.89988036 +0000 UTC m=+101.107938395" Jul 07 14:58:13.922614 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.921656 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5968b76755-zvq54" podStartSLOduration=3.921634334 podStartE2EDuration="3.921634334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:13.920843911 +0000 UTC m=+101.128901955" watchObservedRunningTime="2026-07-07 14:58:13.921634334 +0000 UTC m=+101.129692367" Jul 07 14:58:13.945036 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:13.944971 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-9nghg" podStartSLOduration=1.822785697 podStartE2EDuration="19.944954064s" totalImagesPullingTime="18.122168367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:13.94378484 +0000 UTC m=+101.151842874" watchObservedRunningTime="2026-07-07 14:58:13.944954064 +0000 UTC m=+101.153012098" Jul 07 14:58:15.888145 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:15.888102 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" event={"ID":"5734779e-8983-4e75-9a6e-4787d908cb85","Type":"ContainerStarted","Data":"6206803898cdea2c38741a1628eca89f39efb99039cb8cba5d831d4996cf805d"} Jul 07 14:58:15.888598 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:15.888175 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" event={"ID":"5734779e-8983-4e75-9a6e-4787d908cb85","Type":"ContainerStarted","Data":"b9ff09de9d841cf1e4b50f52f2991085d0aecf09f22ba9b9fa0e1a86b9288d5b"} Jul 07 14:58:15.911365 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:15.911308 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-v6x9v" podStartSLOduration=10.183982129 podStartE2EDuration="11.911285788s" totalImagesPullingTime="1.727303659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:15.909181707 +0000 UTC m=+103.117239765" watchObservedRunningTime="2026-07-07 14:58:15.911285788 +0000 UTC m=+103.119343799" Jul 07 14:58:16.724736 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:16.724699 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mvn5b" Jul 07 14:58:17.647518 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:17.647484 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-696d4545c5-pc6zb"] Jul 07 14:58:17.920647 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:17.920573 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx"] Jul 07 14:58:17.960665 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:17.960633 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:17.961061 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:17.961024 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx"] Jul 07 14:58:17.963747 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:17.963716 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-72d58\"" Jul 07 14:58:17.964660 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:17.964636 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 14:58:17.964779 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:17.964677 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 14:58:17.966455 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:17.966431 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pl58j"] Jul 07 14:58:17.999019 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:17.998994 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.003371 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.003345 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fdkcq\"" Jul 07 14:58:18.003507 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.003383 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 14:58:18.003507 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.003348 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 14:58:18.003741 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.003718 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 14:58:18.038690 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.038655 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6554ecd7-3748-40bf-a291-bd0437e6c973-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.038690 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.038693 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72bf\" (UniqueName: \"kubernetes.io/projected/6554ecd7-3748-40bf-a291-bd0437e6c973-kube-api-access-v72bf\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.038903 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.038768 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-wtmp\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.038903 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.038822 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6554ecd7-3748-40bf-a291-bd0437e6c973-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.038903 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.038862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-tls\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.039065 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.038902 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-accelerators-collector-config\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.039065 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.038929 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trzh6\" (UniqueName: \"kubernetes.io/projected/f5f948de-d5b3-4fa1-a29a-b43803289d62-kube-api-access-trzh6\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.039065 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.038965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6554ecd7-3748-40bf-a291-bd0437e6c973-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.039221 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.039037 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5f948de-d5b3-4fa1-a29a-b43803289d62-root\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.039221 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.039104 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-textfile\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.039221 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.039138 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.039221 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.039178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5f948de-d5b3-4fa1-a29a-b43803289d62-metrics-client-ca\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.039422 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.039224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5f948de-d5b3-4fa1-a29a-b43803289d62-sys\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140290 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-tls\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140290 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140297 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-accelerators-collector-config\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140542 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140321 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trzh6\" (UniqueName: \"kubernetes.io/projected/f5f948de-d5b3-4fa1-a29a-b43803289d62-kube-api-access-trzh6\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140542 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6554ecd7-3748-40bf-a291-bd0437e6c973-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.140542 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5f948de-d5b3-4fa1-a29a-b43803289d62-root\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140542 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140435 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-textfile\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140542 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140465 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140542 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140500 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5f948de-d5b3-4fa1-a29a-b43803289d62-metrics-client-ca\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140542 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140507 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5f948de-d5b3-4fa1-a29a-b43803289d62-root\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140542 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140524 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5f948de-d5b3-4fa1-a29a-b43803289d62-sys\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140919 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140577 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5f948de-d5b3-4fa1-a29a-b43803289d62-sys\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140919 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140591 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6554ecd7-3748-40bf-a291-bd0437e6c973-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.140919 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72bf\" (UniqueName: \"kubernetes.io/projected/6554ecd7-3748-40bf-a291-bd0437e6c973-kube-api-access-v72bf\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.140919 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140679 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-wtmp\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.140919 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140724 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6554ecd7-3748-40bf-a291-bd0437e6c973-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.140919 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140834 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-textfile\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.141245 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.140988 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-accelerators-collector-config\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.141245 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.141087 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5f948de-d5b3-4fa1-a29a-b43803289d62-metrics-client-ca\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.141245 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.141156 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-wtmp\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.141518 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.141500 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6554ecd7-3748-40bf-a291-bd0437e6c973-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.143371 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.143343 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-tls\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.143562 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.143542 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6554ecd7-3748-40bf-a291-bd0437e6c973-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.143635 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.143586 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5f948de-d5b3-4fa1-a29a-b43803289d62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.143690 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.143639 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6554ecd7-3748-40bf-a291-bd0437e6c973-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.152922 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.152880 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72bf\" (UniqueName: \"kubernetes.io/projected/6554ecd7-3748-40bf-a291-bd0437e6c973-kube-api-access-v72bf\") pod \"openshift-state-metrics-54cf96dfcf-68fkx\" (UID: \"6554ecd7-3748-40bf-a291-bd0437e6c973\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.153382 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.153357 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trzh6\" (UniqueName: \"kubernetes.io/projected/f5f948de-d5b3-4fa1-a29a-b43803289d62-kube-api-access-trzh6\") pod \"node-exporter-pl58j\" (UID: \"f5f948de-d5b3-4fa1-a29a-b43803289d62\") " pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.272677 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.272634 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" Jul 07 14:58:18.310132 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.309593 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pl58j" Jul 07 14:58:18.348748 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:58:18.348715 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f948de_d5b3_4fa1_a29a_b43803289d62.slice/crio-f60ea50b0f9ece83c1ad6dc7c95a7e41a92e418509143eefd6f23c47360dcb09 WatchSource:0}: Error finding container f60ea50b0f9ece83c1ad6dc7c95a7e41a92e418509143eefd6f23c47360dcb09: Status 404 returned error can't find the container with id f60ea50b0f9ece83c1ad6dc7c95a7e41a92e418509143eefd6f23c47360dcb09 Jul 07 14:58:18.465743 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.465653 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx"] Jul 07 14:58:18.468994 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:58:18.468932 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6554ecd7_3748_40bf_a291_bd0437e6c973.slice/crio-2128a80c619b5357e0a1b0e3c1291363f2f6fb030f02a5fcc505cdf4de4d2fab WatchSource:0}: Error finding container 2128a80c619b5357e0a1b0e3c1291363f2f6fb030f02a5fcc505cdf4de4d2fab: Status 404 returned error can't find the container with id 2128a80c619b5357e0a1b0e3c1291363f2f6fb030f02a5fcc505cdf4de4d2fab Jul 07 14:58:18.904291 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.904110 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" event={"ID":"6554ecd7-3748-40bf-a291-bd0437e6c973","Type":"ContainerStarted","Data":"580d3173307e9e23010d11f123cbdb2031592f44eeadad59eda5536758ac229e"} Jul 07 14:58:18.904291 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.904189 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" event={"ID":"6554ecd7-3748-40bf-a291-bd0437e6c973","Type":"ContainerStarted","Data":"199f86e434cd280e62adec8c222283be106e0a839c8102724c74aae37c81995d"} Jul 07 14:58:18.904291 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.904238 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" event={"ID":"6554ecd7-3748-40bf-a291-bd0437e6c973","Type":"ContainerStarted","Data":"2128a80c619b5357e0a1b0e3c1291363f2f6fb030f02a5fcc505cdf4de4d2fab"} Jul 07 14:58:18.906316 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:18.906286 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pl58j" event={"ID":"f5f948de-d5b3-4fa1-a29a-b43803289d62","Type":"ContainerStarted","Data":"f60ea50b0f9ece83c1ad6dc7c95a7e41a92e418509143eefd6f23c47360dcb09"} Jul 07 14:58:20.921382 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:20.921298 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" event={"ID":"6554ecd7-3748-40bf-a291-bd0437e6c973","Type":"ContainerStarted","Data":"63e3d4a93f0fa7e31ba05e14d71e44894fde09849dd6af46252e0dd13234c096"} Jul 07 14:58:20.923165 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:20.923135 2566 generic.go:356] "Generic (PLEG): container finished" podID="f5f948de-d5b3-4fa1-a29a-b43803289d62" containerID="d6e7a269b121d7c11aba9fe433e569b0bf9546f7e3a910a5a2448b3c82090bda" exitCode=0 Jul 07 14:58:20.923299 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:20.923214 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pl58j" event={"ID":"f5f948de-d5b3-4fa1-a29a-b43803289d62","Type":"ContainerDied","Data":"d6e7a269b121d7c11aba9fe433e569b0bf9546f7e3a910a5a2448b3c82090bda"} Jul 07 14:58:20.944928 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:20.944866 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-68fkx" podStartSLOduration=2.12434844 podStartE2EDuration="3.944845264s" totalImagesPullingTime="1.820496824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:20.944090934 +0000 UTC m=+108.152148967" watchObservedRunningTime="2026-07-07 14:58:20.944845264 +0000 UTC m=+108.152903297" Jul 07 14:58:21.112918 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.112891 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:21.113070 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.112992 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:21.118516 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.118492 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:21.930817 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.930731 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pl58j" event={"ID":"f5f948de-d5b3-4fa1-a29a-b43803289d62","Type":"ContainerStarted","Data":"a087528bdb6fc4e464c6047b5a3c47dfda692c5ceea935715670bb619d574141"} Jul 07 14:58:21.930817 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.930800 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pl58j" event={"ID":"f5f948de-d5b3-4fa1-a29a-b43803289d62","Type":"ContainerStarted","Data":"d8812d0f0d0a0ea9cbdcbff16aa9753bf494c11459e07293502fad6cc4cc4921"} Jul 07 14:58:21.935491 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.935466 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:58:21.960061 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.959996 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pl58j" podStartSLOduration=2.924955423 podStartE2EDuration="4.959979142s" totalImagesPullingTime="1.934313405s" totalInitContainerRuntime="100.710314ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:21.959443203 +0000 UTC m=+109.167501247" watchObservedRunningTime="2026-07-07 14:58:21.959979142 +0000 UTC m=+109.168037173" Jul 07 14:58:21.985403 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.985374 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:21.985555 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.985426 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:21.990868 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:21.990841 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:22.087281 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:22.087251 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-697c8bdf69-khv5j"] Jul 07 14:58:22.939128 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:22.939088 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:23.148062 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.148009 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm"] Jul 07 14:58:23.172595 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.172560 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm"] Jul 07 14:58:23.172754 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.172663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.177406 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.177381 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 14:58:23.177539 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.177504 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tqmkd\"" Jul 07 14:58:23.178075 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.178036 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 14:58:23.178287 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.178269 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 14:58:23.178461 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.178442 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 14:58:23.178678 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.178658 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 14:58:23.184805 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.184787 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 14:58:23.294298 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.294264 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0737e2-9920-4473-ac78-8844f7e46d21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.294298 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.294302 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-telemeter-client-tls\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.294517 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.294325 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-secret-telemeter-client\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.294517 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.294344 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.294517 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.294380 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-federate-client-tls\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.294517 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.294394 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b0737e2-9920-4473-ac78-8844f7e46d21-metrics-client-ca\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.294517 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.294411 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455md\" (UniqueName: \"kubernetes.io/projected/5b0737e2-9920-4473-ac78-8844f7e46d21-kube-api-access-455md\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.294517 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.294469 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0737e2-9920-4473-ac78-8844f7e46d21-serving-certs-ca-bundle\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.394884 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.394846 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0737e2-9920-4473-ac78-8844f7e46d21-serving-certs-ca-bundle\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.395106 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.394911 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0737e2-9920-4473-ac78-8844f7e46d21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.395106 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.394941 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-telemeter-client-tls\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.395106 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.394965 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-secret-telemeter-client\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.395106 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.394984 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.395345 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.395129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-federate-client-tls\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.395345 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.395172 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b0737e2-9920-4473-ac78-8844f7e46d21-metrics-client-ca\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.395345 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.395205 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455md\" (UniqueName: \"kubernetes.io/projected/5b0737e2-9920-4473-ac78-8844f7e46d21-kube-api-access-455md\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.395763 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.395731 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0737e2-9920-4473-ac78-8844f7e46d21-serving-certs-ca-bundle\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.395998 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.395971 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b0737e2-9920-4473-ac78-8844f7e46d21-metrics-client-ca\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.396093 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.396062 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0737e2-9920-4473-ac78-8844f7e46d21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.397603 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.397576 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-telemeter-client-tls\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.397699 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.397648 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-secret-telemeter-client\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.397846 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.397829 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.397895 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.397876 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5b0737e2-9920-4473-ac78-8844f7e46d21-federate-client-tls\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.406114 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.406095 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-455md\" (UniqueName: \"kubernetes.io/projected/5b0737e2-9920-4473-ac78-8844f7e46d21-kube-api-access-455md\") pod \"telemeter-client-7fb4b857b6-rjfqm\" (UID: \"5b0737e2-9920-4473-ac78-8844f7e46d21\") " pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.481802 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.481777 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" Jul 07 14:58:23.649952 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.649927 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm"] Jul 07 14:58:23.652420 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:58:23.652393 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0737e2_9920_4473_ac78_8844f7e46d21.slice/crio-45cfc789f157573edaccaf939abc47e0138229660f38b9df4f87c58222263bca WatchSource:0}: Error finding container 45cfc789f157573edaccaf939abc47e0138229660f38b9df4f87c58222263bca: Status 404 returned error can't find the container with id 45cfc789f157573edaccaf939abc47e0138229660f38b9df4f87c58222263bca Jul 07 14:58:23.943924 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:23.943829 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" event={"ID":"5b0737e2-9920-4473-ac78-8844f7e46d21","Type":"ContainerStarted","Data":"45cfc789f157573edaccaf939abc47e0138229660f38b9df4f87c58222263bca"} Jul 07 14:58:24.254224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.254191 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:58:24.269242 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.269217 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.272668 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.272642 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 14:58:24.273508 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.273486 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 14:58:24.273620 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.273486 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 14:58:24.273620 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.273531 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 14:58:24.274264 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.274247 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 14:58:24.274332 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.274280 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 14:58:24.274607 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.274590 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 14:58:24.275013 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.274989 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 14:58:24.275143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.275033 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 14:58:24.275143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.275076 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 14:58:24.275143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.274990 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7d70iuc4nbm89\"" Jul 07 14:58:24.275593 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.275573 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 14:58:24.275672 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.275653 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 14:58:24.275866 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.275846 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d7zq5\"" Jul 07 14:58:24.276174 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.276153 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 14:58:24.278826 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.278807 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 14:58:24.285041 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.285015 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:58:24.289172 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.289153 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 14:58:24.406467 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.406699 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406642 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.406699 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406690 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.406835 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406726 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.406835 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.406835 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406817 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.406955 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406864 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhd4d\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-kube-api-access-lhd4d\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.406955 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406907 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.406955 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406942 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config-out\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407076 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.406995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407076 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407061 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407156 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407109 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407156 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407131 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407229 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407162 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407229 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407202 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407306 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407226 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407306 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407252 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-web-config\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407306 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407280 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407422 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.407422 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.407359 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508340 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508242 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508340 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508305 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508374 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508437 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508465 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508523 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508573 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhd4d\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-kube-api-access-lhd4d\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508644 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508671 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config-out\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508701 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508751 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508788 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508816 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508887 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.508916 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508916 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.509465 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.508947 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-web-config\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.509964 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.509909 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.510947 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.510598 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.513316 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.513291 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.513630 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.513607 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.513851 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.513683 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config-out\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.514006 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.513969 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.514236 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.514215 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-web-config\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.515777 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.515431 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.515777 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.515740 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.516101 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.515791 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.520854 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.514715 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.520854 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.518668 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.520854 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.519144 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.520854 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.519874 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.523705 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.522575 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.524317 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.524296 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.524518 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.524493 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.527954 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.527930 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.528885 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.528864 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.530238 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.530186 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhd4d\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-kube-api-access-lhd4d\") pod \"prometheus-k8s-0\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.581319 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.581258 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:24.787448 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.787093 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:58:24.791015 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:58:24.790986 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71f511a_aa5e_4ddb_9fb4_7dc425d85ba5.slice/crio-21b942baf9a4e005fc01a407b74a4fad08284c72b2d53d0e495c7b92aadba5e3 WatchSource:0}: Error finding container 21b942baf9a4e005fc01a407b74a4fad08284c72b2d53d0e495c7b92aadba5e3: Status 404 returned error can't find the container with id 21b942baf9a4e005fc01a407b74a4fad08284c72b2d53d0e495c7b92aadba5e3 Jul 07 14:58:24.948858 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:24.948823 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerStarted","Data":"21b942baf9a4e005fc01a407b74a4fad08284c72b2d53d0e495c7b92aadba5e3"} Jul 07 14:58:26.957540 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:26.957506 2566 generic.go:356] "Generic (PLEG): container finished" podID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerID="1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3" exitCode=0 Jul 07 14:58:26.958019 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:26.957591 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerDied","Data":"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3"} Jul 07 14:58:26.959854 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:26.959824 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" event={"ID":"5b0737e2-9920-4473-ac78-8844f7e46d21","Type":"ContainerStarted","Data":"fe450652d5cb1cca6e13bebb2932f4de920c5a942e39d09df535369494a93abc"} Jul 07 14:58:26.959951 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:26.959878 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" event={"ID":"5b0737e2-9920-4473-ac78-8844f7e46d21","Type":"ContainerStarted","Data":"eacc53fd870a5bd65bd19723d8bc4f693ce9d9e594c8865e3fc073ada05d09f6"} Jul 07 14:58:26.959951 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:26.959899 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" event={"ID":"5b0737e2-9920-4473-ac78-8844f7e46d21","Type":"ContainerStarted","Data":"8035919f2993f5449be7e3361687edebed21a34c3aeaa10791b4d4ae2137ff06"} Jul 07 14:58:27.014355 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:27.014301 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7fb4b857b6-rjfqm" podStartSLOduration=1.2026048679999999 podStartE2EDuration="4.014284779s" totalImagesPullingTime="2.811679911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:27.013444921 +0000 UTC m=+114.221502977" watchObservedRunningTime="2026-07-07 14:58:27.014284779 +0000 UTC m=+114.222342811" Jul 07 14:58:30.978734 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:30.978704 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerStarted","Data":"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724"} Jul 07 14:58:31.985484 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:31.985448 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerStarted","Data":"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974"} Jul 07 14:58:33.998092 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:33.998036 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerStarted","Data":"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03"} Jul 07 14:58:33.998534 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:33.998129 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerStarted","Data":"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3"} Jul 07 14:58:33.998534 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:33.998159 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerStarted","Data":"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda"} Jul 07 14:58:33.998534 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:33.998185 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerStarted","Data":"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7"} Jul 07 14:58:34.050689 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:34.050636 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.42676425 podStartE2EDuration="10.050618658s" totalImagesPullingTime="7.595003965s" totalInitContainerRuntime="28.850443ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:58:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:34.049939776 +0000 UTC m=+121.257997808" watchObservedRunningTime="2026-07-07 14:58:34.050618658 +0000 UTC m=+121.258676736" Jul 07 14:58:34.581504 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:34.581473 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:58:38.848741 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:38.848704 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5968b76755-zvq54"] Jul 07 14:58:40.021339 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:40.021304 2566 generic.go:356] "Generic (PLEG): container finished" podID="171153e3-a823-4d76-a913-f83499139e27" containerID="7924f98de40f3462691a7b54f44dfc700bfae2927abb936132946a9eaa2159a6" exitCode=0 Jul 07 14:58:40.021851 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:40.021375 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" event={"ID":"171153e3-a823-4d76-a913-f83499139e27","Type":"ContainerDied","Data":"7924f98de40f3462691a7b54f44dfc700bfae2927abb936132946a9eaa2159a6"} Jul 07 14:58:40.021851 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:40.021730 2566 scope.go:117] "RemoveContainer" containerID="7924f98de40f3462691a7b54f44dfc700bfae2927abb936132946a9eaa2159a6" Jul 07 14:58:41.027403 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:41.027366 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-kqd56" event={"ID":"171153e3-a823-4d76-a913-f83499139e27","Type":"ContainerStarted","Data":"fea2710eb298cc7510d4ab08ac02a2c1c3336361a40f6ed24b44320f39231626"} Jul 07 14:58:42.672636 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.672593 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" podUID="02236daa-01c6-4fcf-b497-444d8258a20b" containerName="registry" containerID="cri-o://de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b" gracePeriod=30 Jul 07 14:58:42.703705 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.703675 2566 patch_prober.go:28] interesting pod/image-registry-696d4545c5-pc6zb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" start-of-body= Jul 07 14:58:42.703801 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.703737 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" podUID="02236daa-01c6-4fcf-b497-444d8258a20b" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" Jul 07 14:58:42.952154 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.952086 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:58:42.982138 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982104 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02236daa-01c6-4fcf-b497-444d8258a20b-ca-trust-extracted\") pod \"02236daa-01c6-4fcf-b497-444d8258a20b\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " Jul 07 14:58:42.982138 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982142 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-trusted-ca\") pod \"02236daa-01c6-4fcf-b497-444d8258a20b\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " Jul 07 14:58:42.982359 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982176 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-installation-pull-secrets\") pod \"02236daa-01c6-4fcf-b497-444d8258a20b\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " Jul 07 14:58:42.982359 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982195 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") pod \"02236daa-01c6-4fcf-b497-444d8258a20b\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " Jul 07 14:58:42.982359 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982218 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t5kt\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-kube-api-access-8t5kt\") pod \"02236daa-01c6-4fcf-b497-444d8258a20b\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " Jul 07 14:58:42.982359 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982258 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-image-registry-private-configuration\") pod \"02236daa-01c6-4fcf-b497-444d8258a20b\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " Jul 07 14:58:42.982359 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982282 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-bound-sa-token\") pod \"02236daa-01c6-4fcf-b497-444d8258a20b\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " Jul 07 14:58:42.982359 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982329 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-registry-certificates\") pod \"02236daa-01c6-4fcf-b497-444d8258a20b\" (UID: \"02236daa-01c6-4fcf-b497-444d8258a20b\") " Jul 07 14:58:42.983257 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982963 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "02236daa-01c6-4fcf-b497-444d8258a20b" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:58:42.983257 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.982985 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "02236daa-01c6-4fcf-b497-444d8258a20b" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:58:42.985599 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.985551 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "02236daa-01c6-4fcf-b497-444d8258a20b" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:58:42.985699 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.985605 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-kube-api-access-8t5kt" (OuterVolumeSpecName: "kube-api-access-8t5kt") pod "02236daa-01c6-4fcf-b497-444d8258a20b" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b"). InnerVolumeSpecName "kube-api-access-8t5kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:58:42.985822 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.985779 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "02236daa-01c6-4fcf-b497-444d8258a20b" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:58:42.985911 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.985884 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "02236daa-01c6-4fcf-b497-444d8258a20b" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:58:42.985970 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.985952 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "02236daa-01c6-4fcf-b497-444d8258a20b" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:58:42.991032 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:42.991007 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02236daa-01c6-4fcf-b497-444d8258a20b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "02236daa-01c6-4fcf-b497-444d8258a20b" (UID: "02236daa-01c6-4fcf-b497-444d8258a20b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:58:43.037105 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.037068 2566 generic.go:356] "Generic (PLEG): container finished" podID="02236daa-01c6-4fcf-b497-444d8258a20b" containerID="de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b" exitCode=0 Jul 07 14:58:43.037240 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.037120 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" Jul 07 14:58:43.037240 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.037122 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" event={"ID":"02236daa-01c6-4fcf-b497-444d8258a20b","Type":"ContainerDied","Data":"de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b"} Jul 07 14:58:43.037366 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.037248 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-696d4545c5-pc6zb" event={"ID":"02236daa-01c6-4fcf-b497-444d8258a20b","Type":"ContainerDied","Data":"4d2c2527b09f80f6a96d7b7025fd55ec09f6bbd4e7c455e1a7ee4f4fc0da39bd"} Jul 07 14:58:43.037366 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.037278 2566 scope.go:117] "RemoveContainer" containerID="de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b" Jul 07 14:58:43.045900 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.045880 2566 scope.go:117] "RemoveContainer" containerID="de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b" Jul 07 14:58:43.046193 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:58:43.046170 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b\": container with ID starting with de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b not found: ID does not exist" containerID="de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b" Jul 07 14:58:43.046280 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.046204 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b"} err="failed to get container status \"de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b\": rpc error: code = NotFound desc = could not find container \"de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b\": container with ID starting with de5a5f9e0db55eedc2da83729381d053461a06c2c347accf3f00276ce7951b4b not found: ID does not exist" Jul 07 14:58:43.058562 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.058537 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-696d4545c5-pc6zb"] Jul 07 14:58:43.064123 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.064099 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-696d4545c5-pc6zb"] Jul 07 14:58:43.083563 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.083540 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:58:43.083652 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.083636 2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-bound-sa-token\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:43.083690 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.083659 2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-registry-certificates\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:43.083690 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.083672 2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02236daa-01c6-4fcf-b497-444d8258a20b-ca-trust-extracted\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:43.083690 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.083686 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02236daa-01c6-4fcf-b497-444d8258a20b-trusted-ca\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:43.083784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.083698 2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-installation-pull-secrets\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:43.083784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.083710 2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-registry-tls\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:43.083784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.083723 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5kt\" (UniqueName: \"kubernetes.io/projected/02236daa-01c6-4fcf-b497-444d8258a20b-kube-api-access-8t5kt\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:43.083784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.083739 2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02236daa-01c6-4fcf-b497-444d8258a20b-image-registry-private-configuration\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:43.085786 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.085770 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f115fbf4-80c0-4991-b663-61ce83d6e968-metrics-certs\") pod \"network-metrics-daemon-jw459\" (UID: \"f115fbf4-80c0-4991-b663-61ce83d6e968\") " pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:58:43.229547 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.229459 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nm9h4\"" Jul 07 14:58:43.237274 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.237254 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw459" Jul 07 14:58:43.316974 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.316936 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02236daa-01c6-4fcf-b497-444d8258a20b" path="/var/lib/kubelet/pods/02236daa-01c6-4fcf-b497-444d8258a20b/volumes" Jul 07 14:58:43.406037 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:43.406004 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jw459"] Jul 07 14:58:43.408241 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:58:43.408215 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf115fbf4_80c0_4991_b663_61ce83d6e968.slice/crio-bdb4d9ff32b4ad57b83bf290ac2e2ceef4d1dfac126b5fba8e9474a047094872 WatchSource:0}: Error finding container bdb4d9ff32b4ad57b83bf290ac2e2ceef4d1dfac126b5fba8e9474a047094872: Status 404 returned error can't find the container with id bdb4d9ff32b4ad57b83bf290ac2e2ceef4d1dfac126b5fba8e9474a047094872 Jul 07 14:58:44.043668 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:44.043630 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jw459" event={"ID":"f115fbf4-80c0-4991-b663-61ce83d6e968","Type":"ContainerStarted","Data":"bdb4d9ff32b4ad57b83bf290ac2e2ceef4d1dfac126b5fba8e9474a047094872"} Jul 07 14:58:46.055360 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:46.055325 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jw459" event={"ID":"f115fbf4-80c0-4991-b663-61ce83d6e968","Type":"ContainerStarted","Data":"d53632e2a6fdad268c5e57949ba987fa88aa59574f6230d31d9b9849bf4cf93b"} Jul 07 14:58:46.055918 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:46.055383 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jw459" event={"ID":"f115fbf4-80c0-4991-b663-61ce83d6e968","Type":"ContainerStarted","Data":"7270d0314f8552390378b9beb3efcd7180f8720171e94e352379c006368c3234"} Jul 07 14:58:46.076062 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:46.075994 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jw459" podStartSLOduration=131.499525912 podStartE2EDuration="2m13.075972628s" totalImagesPullingTime="1.576446716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:58:46.074873213 +0000 UTC m=+133.282931246" watchObservedRunningTime="2026-07-07 14:58:46.075972628 +0000 UTC m=+133.284030662" Jul 07 14:58:48.966527 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:48.966469 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-697c8bdf69-khv5j" podUID="ee76827a-c9a7-4851-9e41-a268db8c3043" containerName="console" containerID="cri-o://efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75" gracePeriod=15 Jul 07 14:58:49.238330 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.238259 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:49.337621 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.337587 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-oauth-config\") pod \"ee76827a-c9a7-4851-9e41-a268db8c3043\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " Jul 07 14:58:49.337784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.337640 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-console-config\") pod \"ee76827a-c9a7-4851-9e41-a268db8c3043\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " Jul 07 14:58:49.337784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.337662 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzrg4\" (UniqueName: \"kubernetes.io/projected/ee76827a-c9a7-4851-9e41-a268db8c3043-kube-api-access-hzrg4\") pod \"ee76827a-c9a7-4851-9e41-a268db8c3043\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " Jul 07 14:58:49.337784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.337678 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-service-ca\") pod \"ee76827a-c9a7-4851-9e41-a268db8c3043\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " Jul 07 14:58:49.337784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.337701 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-serving-cert\") pod \"ee76827a-c9a7-4851-9e41-a268db8c3043\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " Jul 07 14:58:49.337784 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.337717 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-oauth-serving-cert\") pod \"ee76827a-c9a7-4851-9e41-a268db8c3043\" (UID: \"ee76827a-c9a7-4851-9e41-a268db8c3043\") " Jul 07 14:58:49.338213 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.338174 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-service-ca" (OuterVolumeSpecName: "service-ca") pod "ee76827a-c9a7-4851-9e41-a268db8c3043" (UID: "ee76827a-c9a7-4851-9e41-a268db8c3043"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:58:49.338213 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.338191 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-console-config" (OuterVolumeSpecName: "console-config") pod "ee76827a-c9a7-4851-9e41-a268db8c3043" (UID: "ee76827a-c9a7-4851-9e41-a268db8c3043"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:58:49.338360 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.338239 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ee76827a-c9a7-4851-9e41-a268db8c3043" (UID: "ee76827a-c9a7-4851-9e41-a268db8c3043"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:58:49.340034 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.340005 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ee76827a-c9a7-4851-9e41-a268db8c3043" (UID: "ee76827a-c9a7-4851-9e41-a268db8c3043"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:58:49.340225 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.340079 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ee76827a-c9a7-4851-9e41-a268db8c3043" (UID: "ee76827a-c9a7-4851-9e41-a268db8c3043"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:58:49.340225 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.340106 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee76827a-c9a7-4851-9e41-a268db8c3043-kube-api-access-hzrg4" (OuterVolumeSpecName: "kube-api-access-hzrg4") pod "ee76827a-c9a7-4851-9e41-a268db8c3043" (UID: "ee76827a-c9a7-4851-9e41-a268db8c3043"). InnerVolumeSpecName "kube-api-access-hzrg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:58:49.439399 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.439360 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-serving-cert\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:49.439399 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.439391 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-oauth-serving-cert\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:49.439399 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.439400 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee76827a-c9a7-4851-9e41-a268db8c3043-console-oauth-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:49.439399 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.439408 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-console-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:49.439663 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.439418 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzrg4\" (UniqueName: \"kubernetes.io/projected/ee76827a-c9a7-4851-9e41-a268db8c3043-kube-api-access-hzrg4\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:49.439663 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:49.439427 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee76827a-c9a7-4851-9e41-a268db8c3043-service-ca\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:58:50.079141 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:50.079100 2566 generic.go:356] "Generic (PLEG): container finished" podID="ee76827a-c9a7-4851-9e41-a268db8c3043" containerID="efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75" exitCode=0 Jul 07 14:58:50.079603 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:50.079150 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697c8bdf69-khv5j" Jul 07 14:58:50.079603 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:50.079188 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697c8bdf69-khv5j" event={"ID":"ee76827a-c9a7-4851-9e41-a268db8c3043","Type":"ContainerDied","Data":"efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75"} Jul 07 14:58:50.079603 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:50.079254 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697c8bdf69-khv5j" event={"ID":"ee76827a-c9a7-4851-9e41-a268db8c3043","Type":"ContainerDied","Data":"4d798c684739c41385a3bd098154e091158f3114ba8cc3befc70efd6213af470"} Jul 07 14:58:50.079603 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:50.079286 2566 scope.go:117] "RemoveContainer" containerID="efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75" Jul 07 14:58:50.087660 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:50.087644 2566 scope.go:117] "RemoveContainer" containerID="efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75" Jul 07 14:58:50.087905 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:58:50.087888 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75\": container with ID starting with efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75 not found: ID does not exist" containerID="efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75" Jul 07 14:58:50.087961 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:50.087914 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75"} err="failed to get container status \"efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75\": rpc error: code = NotFound desc = could not find container \"efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75\": container with ID starting with efd83e71500dd0cb7ceea283d0b34c44dc002b97a538f1508400f6013d48ef75 not found: ID does not exist" Jul 07 14:58:50.113288 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:50.113260 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-697c8bdf69-khv5j"] Jul 07 14:58:50.120024 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:50.120000 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-697c8bdf69-khv5j"] Jul 07 14:58:51.310762 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:58:51.310733 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee76827a-c9a7-4851-9e41-a268db8c3043" path="/var/lib/kubelet/pods/ee76827a-c9a7-4851-9e41-a268db8c3043/volumes" Jul 07 14:59:03.868831 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:03.868770 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5968b76755-zvq54" podUID="fbff8bbe-6289-484e-a99c-38c0af9ff6a9" containerName="console" containerID="cri-o://16369ad04d1c570fb40ebaa867e94e40153774919540c33baa6d1387c32c381f" gracePeriod=15 Jul 07 14:59:04.133959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.133894 2566 generic.go:356] "Generic (PLEG): container finished" podID="fbff8bbe-6289-484e-a99c-38c0af9ff6a9" containerID="16369ad04d1c570fb40ebaa867e94e40153774919540c33baa6d1387c32c381f" exitCode=0 Jul 07 14:59:04.133959 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.133930 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5968b76755-zvq54" event={"ID":"fbff8bbe-6289-484e-a99c-38c0af9ff6a9","Type":"ContainerDied","Data":"16369ad04d1c570fb40ebaa867e94e40153774919540c33baa6d1387c32c381f"} Jul 07 14:59:04.134140 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.133965 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5968b76755-zvq54" event={"ID":"fbff8bbe-6289-484e-a99c-38c0af9ff6a9","Type":"ContainerDied","Data":"66bd7f22a9d19ad0a7f44ed2d9542d7e5a309a886a91c6cd3b451f9043a32a78"} Jul 07 14:59:04.134140 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.133980 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66bd7f22a9d19ad0a7f44ed2d9542d7e5a309a886a91c6cd3b451f9043a32a78" Jul 07 14:59:04.149119 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.149085 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:59:04.274642 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.274604 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-serving-cert\") pod \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " Jul 07 14:59:04.274833 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.274709 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-config\") pod \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " Jul 07 14:59:04.274833 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.274737 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-service-ca\") pod \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " Jul 07 14:59:04.274833 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.274761 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxs7k\" (UniqueName: \"kubernetes.io/projected/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-kube-api-access-lxs7k\") pod \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " Jul 07 14:59:04.274833 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.274794 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-oauth-config\") pod \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " Jul 07 14:59:04.275068 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.274845 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-trusted-ca-bundle\") pod \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " Jul 07 14:59:04.275612 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.275467 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-config" (OuterVolumeSpecName: "console-config") pod "fbff8bbe-6289-484e-a99c-38c0af9ff6a9" (UID: "fbff8bbe-6289-484e-a99c-38c0af9ff6a9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:04.275612 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.275591 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-service-ca" (OuterVolumeSpecName: "service-ca") pod "fbff8bbe-6289-484e-a99c-38c0af9ff6a9" (UID: "fbff8bbe-6289-484e-a99c-38c0af9ff6a9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:04.275814 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.275662 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fbff8bbe-6289-484e-a99c-38c0af9ff6a9" (UID: "fbff8bbe-6289-484e-a99c-38c0af9ff6a9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:04.275814 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.275694 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-oauth-serving-cert\") pod \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\" (UID: \"fbff8bbe-6289-484e-a99c-38c0af9ff6a9\") " Jul 07 14:59:04.277450 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.276538 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:04.277450 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.276579 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-service-ca\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:04.277450 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.276595 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-trusted-ca-bundle\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:04.281031 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.277814 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fbff8bbe-6289-484e-a99c-38c0af9ff6a9" (UID: "fbff8bbe-6289-484e-a99c-38c0af9ff6a9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:04.281031 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.277910 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fbff8bbe-6289-484e-a99c-38c0af9ff6a9" (UID: "fbff8bbe-6289-484e-a99c-38c0af9ff6a9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:04.281031 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.278109 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-kube-api-access-lxs7k" (OuterVolumeSpecName: "kube-api-access-lxs7k") pod "fbff8bbe-6289-484e-a99c-38c0af9ff6a9" (UID: "fbff8bbe-6289-484e-a99c-38c0af9ff6a9"). InnerVolumeSpecName "kube-api-access-lxs7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:59:04.281502 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.281470 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fbff8bbe-6289-484e-a99c-38c0af9ff6a9" (UID: "fbff8bbe-6289-484e-a99c-38c0af9ff6a9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:04.377765 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.377734 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-oauth-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:04.377765 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.377760 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-oauth-serving-cert\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:04.377765 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.377769 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-console-serving-cert\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:04.378025 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:04.377778 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxs7k\" (UniqueName: \"kubernetes.io/projected/fbff8bbe-6289-484e-a99c-38c0af9ff6a9-kube-api-access-lxs7k\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:05.137367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:05.137333 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5968b76755-zvq54" Jul 07 14:59:05.161993 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:05.161960 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5968b76755-zvq54"] Jul 07 14:59:05.165792 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:05.165767 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5968b76755-zvq54"] Jul 07 14:59:05.312686 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:05.312652 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbff8bbe-6289-484e-a99c-38c0af9ff6a9" path="/var/lib/kubelet/pods/fbff8bbe-6289-484e-a99c-38c0af9ff6a9/volumes" Jul 07 14:59:24.581994 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:24.581950 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:24.601419 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:24.601390 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:25.230765 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:25.230694 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:42.682698 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.682489 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:59:42.683262 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.682884 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="prometheus" containerID="cri-o://52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" gracePeriod=600 Jul 07 14:59:42.683348 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.683237 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy-web" containerID="cri-o://82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" gracePeriod=600 Jul 07 14:59:42.683348 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.683313 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" gracePeriod=600 Jul 07 14:59:42.683451 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.683242 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy" containerID="cri-o://7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" gracePeriod=600 Jul 07 14:59:42.683852 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.683236 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="thanos-sidecar" containerID="cri-o://74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" gracePeriod=600 Jul 07 14:59:42.683852 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.683407 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="config-reloader" containerID="cri-o://f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" gracePeriod=600 Jul 07 14:59:42.937761 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.937697 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:42.974229 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974195 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-serving-certs-ca-bundle\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974427 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974252 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-grpc-tls\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974427 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974284 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-trusted-ca-bundle\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974427 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974323 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974427 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974353 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-web-config\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974427 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974384 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-tls\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974427 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974422 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-kubelet-serving-ca-bundle\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974742 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974450 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-1\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974742 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974487 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-2\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974742 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974516 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974742 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974548 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-db\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974742 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974600 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-metrics-client-ca\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974742 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974634 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config-out\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974742 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974635 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:42.974742 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974671 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-thanos-prometheus-http-client-file\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.974742 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974704 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974744 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-0\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974760 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974788 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhd4d\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-kube-api-access-lhd4d\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974817 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-tls-assets\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974846 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-kube-rbac-proxy\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974863 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.974878 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-metrics-client-certs\") pod \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\" (UID: \"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5\") " Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.975039 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.975188 2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.975209 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.975224 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.975239 2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-metrics-client-ca\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:42.975353 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.975287 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:42.976863 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.976828 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:59:42.978394 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.978359 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:42.978622 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.978576 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:42.978798 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.978765 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:59:42.979151 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.979124 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:42.981181 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.981140 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:42.981326 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.981293 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:42.981705 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.981676 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:59:42.981849 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.981807 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config" (OuterVolumeSpecName: "config") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:42.981931 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.981866 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:42.981931 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.981912 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:42.982104 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.981996 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-kube-api-access-lhd4d" (OuterVolumeSpecName: "kube-api-access-lhd4d") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "kube-api-access-lhd4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:59:42.982104 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.982088 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config-out" (OuterVolumeSpecName: "config-out") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:59:42.983928 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.983901 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:42.991926 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:42.991902 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-web-config" (OuterVolumeSpecName: "web-config") pod "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" (UID: "b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:59:43.076127 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076092 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076127 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076119 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhd4d\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-kube-api-access-lhd4d\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076127 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076129 2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-tls-assets\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076139 2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-kube-rbac-proxy\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076148 2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-metrics-client-certs\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076157 2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-grpc-tls\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076167 2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076176 2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-web-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076184 2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076192 2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076202 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076210 2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076217 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-prometheus-k8s-db\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076228 2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-config-out\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076236 2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.076344 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.076245 2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 14:59:43.288819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.288785 2566 generic.go:356] "Generic (PLEG): container finished" podID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerID="83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" exitCode=0 Jul 07 14:59:43.288819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.288815 2566 generic.go:356] "Generic (PLEG): container finished" podID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerID="7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" exitCode=0 Jul 07 14:59:43.288819 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.288824 2566 generic.go:356] "Generic (PLEG): container finished" podID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerID="82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" exitCode=0 Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.288833 2566 generic.go:356] "Generic (PLEG): container finished" podID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerID="74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" exitCode=0 Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.288842 2566 generic.go:356] "Generic (PLEG): container finished" podID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerID="f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" exitCode=0 Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.288850 2566 generic.go:356] "Generic (PLEG): container finished" podID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerID="52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" exitCode=0 Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.288862 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.288860 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerDied","Data":"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03"} Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.289018 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerDied","Data":"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3"} Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.289067 2566 scope.go:117] "RemoveContainer" containerID="83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.289079 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerDied","Data":"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda"} Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.289114 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerDied","Data":"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7"} Jul 07 14:59:43.289139 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.289144 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerDied","Data":"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974"} Jul 07 14:59:43.289590 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.289177 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerDied","Data":"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724"} Jul 07 14:59:43.289590 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.289214 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5","Type":"ContainerDied","Data":"21b942baf9a4e005fc01a407b74a4fad08284c72b2d53d0e495c7b92aadba5e3"} Jul 07 14:59:43.298081 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.297850 2566 scope.go:117] "RemoveContainer" containerID="7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" Jul 07 14:59:43.304678 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.304663 2566 scope.go:117] "RemoveContainer" containerID="82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" Jul 07 14:59:43.311096 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.311071 2566 scope.go:117] "RemoveContainer" containerID="74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" Jul 07 14:59:43.315718 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.315697 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:59:43.319251 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.318492 2566 scope.go:117] "RemoveContainer" containerID="f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" Jul 07 14:59:43.321000 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.320981 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:59:43.325367 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.325352 2566 scope.go:117] "RemoveContainer" containerID="52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" Jul 07 14:59:43.331976 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.331955 2566 scope.go:117] "RemoveContainer" containerID="1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3" Jul 07 14:59:43.338413 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.338386 2566 scope.go:117] "RemoveContainer" containerID="83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" Jul 07 14:59:43.338664 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:59:43.338644 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": container with ID starting with 83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03 not found: ID does not exist" containerID="83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" Jul 07 14:59:43.338719 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.338675 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03"} err="failed to get container status \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": rpc error: code = NotFound desc = could not find container \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": container with ID starting with 83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03 not found: ID does not exist" Jul 07 14:59:43.338719 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.338694 2566 scope.go:117] "RemoveContainer" containerID="7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" Jul 07 14:59:43.338954 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:59:43.338934 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": container with ID starting with 7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3 not found: ID does not exist" containerID="7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" Jul 07 14:59:43.339088 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.338960 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3"} err="failed to get container status \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": rpc error: code = NotFound desc = could not find container \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": container with ID starting with 7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3 not found: ID does not exist" Jul 07 14:59:43.339088 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.338982 2566 scope.go:117] "RemoveContainer" containerID="82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" Jul 07 14:59:43.339301 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:59:43.339279 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": container with ID starting with 82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda not found: ID does not exist" containerID="82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" Jul 07 14:59:43.339343 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.339310 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda"} err="failed to get container status \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": rpc error: code = NotFound desc = could not find container \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": container with ID starting with 82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda not found: ID does not exist" Jul 07 14:59:43.339343 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.339331 2566 scope.go:117] "RemoveContainer" containerID="74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" Jul 07 14:59:43.339590 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:59:43.339570 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": container with ID starting with 74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7 not found: ID does not exist" containerID="74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" Jul 07 14:59:43.339666 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.339594 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7"} err="failed to get container status \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": rpc error: code = NotFound desc = could not find container \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": container with ID starting with 74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7 not found: ID does not exist" Jul 07 14:59:43.339666 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.339608 2566 scope.go:117] "RemoveContainer" containerID="f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" Jul 07 14:59:43.339882 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:59:43.339860 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": container with ID starting with f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974 not found: ID does not exist" containerID="f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" Jul 07 14:59:43.339940 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.339890 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974"} err="failed to get container status \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": rpc error: code = NotFound desc = could not find container \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": container with ID starting with f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974 not found: ID does not exist" Jul 07 14:59:43.339940 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.339906 2566 scope.go:117] "RemoveContainer" containerID="52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" Jul 07 14:59:43.340200 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:59:43.340181 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": container with ID starting with 52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724 not found: ID does not exist" containerID="52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" Jul 07 14:59:43.340292 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.340202 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724"} err="failed to get container status \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": rpc error: code = NotFound desc = could not find container \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": container with ID starting with 52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724 not found: ID does not exist" Jul 07 14:59:43.340292 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.340217 2566 scope.go:117] "RemoveContainer" containerID="1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3" Jul 07 14:59:43.340493 ip-10-0-130-13 kubenswrapper[2566]: E0707 14:59:43.340476 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": container with ID starting with 1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3 not found: ID does not exist" containerID="1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3" Jul 07 14:59:43.340538 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.340502 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3"} err="failed to get container status \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": rpc error: code = NotFound desc = could not find container \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": container with ID starting with 1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3 not found: ID does not exist" Jul 07 14:59:43.340538 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.340519 2566 scope.go:117] "RemoveContainer" containerID="83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" Jul 07 14:59:43.340746 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.340731 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03"} err="failed to get container status \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": rpc error: code = NotFound desc = could not find container \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": container with ID starting with 83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03 not found: ID does not exist" Jul 07 14:59:43.340787 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.340746 2566 scope.go:117] "RemoveContainer" containerID="7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" Jul 07 14:59:43.340971 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.340955 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3"} err="failed to get container status \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": rpc error: code = NotFound desc = could not find container \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": container with ID starting with 7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3 not found: ID does not exist" Jul 07 14:59:43.341011 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.340971 2566 scope.go:117] "RemoveContainer" containerID="82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" Jul 07 14:59:43.341211 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.341189 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda"} err="failed to get container status \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": rpc error: code = NotFound desc = could not find container \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": container with ID starting with 82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda not found: ID does not exist" Jul 07 14:59:43.341265 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.341211 2566 scope.go:117] "RemoveContainer" containerID="74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" Jul 07 14:59:43.341419 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.341402 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7"} err="failed to get container status \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": rpc error: code = NotFound desc = could not find container \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": container with ID starting with 74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7 not found: ID does not exist" Jul 07 14:59:43.341459 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.341420 2566 scope.go:117] "RemoveContainer" containerID="f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" Jul 07 14:59:43.341630 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.341614 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974"} err="failed to get container status \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": rpc error: code = NotFound desc = could not find container \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": container with ID starting with f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974 not found: ID does not exist" Jul 07 14:59:43.341676 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.341630 2566 scope.go:117] "RemoveContainer" containerID="52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" Jul 07 14:59:43.341821 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.341806 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724"} err="failed to get container status \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": rpc error: code = NotFound desc = could not find container \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": container with ID starting with 52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724 not found: ID does not exist" Jul 07 14:59:43.341857 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.341822 2566 scope.go:117] "RemoveContainer" containerID="1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3" Jul 07 14:59:43.342030 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342011 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3"} err="failed to get container status \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": rpc error: code = NotFound desc = could not find container \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": container with ID starting with 1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3 not found: ID does not exist" Jul 07 14:59:43.342099 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342030 2566 scope.go:117] "RemoveContainer" containerID="83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" Jul 07 14:59:43.342273 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342257 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03"} err="failed to get container status \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": rpc error: code = NotFound desc = could not find container \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": container with ID starting with 83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03 not found: ID does not exist" Jul 07 14:59:43.342325 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342274 2566 scope.go:117] "RemoveContainer" containerID="7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" Jul 07 14:59:43.342474 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342459 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3"} err="failed to get container status \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": rpc error: code = NotFound desc = could not find container \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": container with ID starting with 7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3 not found: ID does not exist" Jul 07 14:59:43.342474 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342473 2566 scope.go:117] "RemoveContainer" containerID="82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" Jul 07 14:59:43.342652 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342636 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda"} err="failed to get container status \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": rpc error: code = NotFound desc = could not find container \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": container with ID starting with 82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda not found: ID does not exist" Jul 07 14:59:43.342691 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342653 2566 scope.go:117] "RemoveContainer" containerID="74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" Jul 07 14:59:43.342848 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342833 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7"} err="failed to get container status \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": rpc error: code = NotFound desc = could not find container \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": container with ID starting with 74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7 not found: ID does not exist" Jul 07 14:59:43.342893 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.342849 2566 scope.go:117] "RemoveContainer" containerID="f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" Jul 07 14:59:43.343100 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343059 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974"} err="failed to get container status \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": rpc error: code = NotFound desc = could not find container \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": container with ID starting with f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974 not found: ID does not exist" Jul 07 14:59:43.343100 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343073 2566 scope.go:117] "RemoveContainer" containerID="52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" Jul 07 14:59:43.343258 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343236 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724"} err="failed to get container status \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": rpc error: code = NotFound desc = could not find container \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": container with ID starting with 52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724 not found: ID does not exist" Jul 07 14:59:43.343295 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343260 2566 scope.go:117] "RemoveContainer" containerID="1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3" Jul 07 14:59:43.343462 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343447 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3"} err="failed to get container status \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": rpc error: code = NotFound desc = could not find container \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": container with ID starting with 1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3 not found: ID does not exist" Jul 07 14:59:43.343517 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343463 2566 scope.go:117] "RemoveContainer" containerID="83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" Jul 07 14:59:43.343683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343667 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03"} err="failed to get container status \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": rpc error: code = NotFound desc = could not find container \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": container with ID starting with 83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03 not found: ID does not exist" Jul 07 14:59:43.343683 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343683 2566 scope.go:117] "RemoveContainer" containerID="7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" Jul 07 14:59:43.343897 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343882 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3"} err="failed to get container status \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": rpc error: code = NotFound desc = could not find container \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": container with ID starting with 7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3 not found: ID does not exist" Jul 07 14:59:43.343940 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.343898 2566 scope.go:117] "RemoveContainer" containerID="82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" Jul 07 14:59:43.344180 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.344158 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda"} err="failed to get container status \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": rpc error: code = NotFound desc = could not find container \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": container with ID starting with 82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda not found: ID does not exist" Jul 07 14:59:43.344245 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.344181 2566 scope.go:117] "RemoveContainer" containerID="74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" Jul 07 14:59:43.344386 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.344370 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7"} err="failed to get container status \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": rpc error: code = NotFound desc = could not find container \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": container with ID starting with 74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7 not found: ID does not exist" Jul 07 14:59:43.344428 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.344387 2566 scope.go:117] "RemoveContainer" containerID="f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" Jul 07 14:59:43.344613 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.344592 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974"} err="failed to get container status \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": rpc error: code = NotFound desc = could not find container \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": container with ID starting with f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974 not found: ID does not exist" Jul 07 14:59:43.344613 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.344612 2566 scope.go:117] "RemoveContainer" containerID="52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" Jul 07 14:59:43.344860 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.344833 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724"} err="failed to get container status \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": rpc error: code = NotFound desc = could not find container \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": container with ID starting with 52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724 not found: ID does not exist" Jul 07 14:59:43.344860 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.344854 2566 scope.go:117] "RemoveContainer" containerID="1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3" Jul 07 14:59:43.345171 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.345140 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3"} err="failed to get container status \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": rpc error: code = NotFound desc = could not find container \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": container with ID starting with 1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3 not found: ID does not exist" Jul 07 14:59:43.345171 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.345168 2566 scope.go:117] "RemoveContainer" containerID="83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" Jul 07 14:59:43.345527 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.345477 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03"} err="failed to get container status \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": rpc error: code = NotFound desc = could not find container \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": container with ID starting with 83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03 not found: ID does not exist" Jul 07 14:59:43.345639 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.345620 2566 scope.go:117] "RemoveContainer" containerID="7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" Jul 07 14:59:43.346011 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.345990 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3"} err="failed to get container status \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": rpc error: code = NotFound desc = could not find container \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": container with ID starting with 7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3 not found: ID does not exist" Jul 07 14:59:43.346011 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.346011 2566 scope.go:117] "RemoveContainer" containerID="82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" Jul 07 14:59:43.346299 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.346281 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda"} err="failed to get container status \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": rpc error: code = NotFound desc = could not find container \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": container with ID starting with 82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda not found: ID does not exist" Jul 07 14:59:43.346378 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.346300 2566 scope.go:117] "RemoveContainer" containerID="74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" Jul 07 14:59:43.346551 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.346526 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7"} err="failed to get container status \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": rpc error: code = NotFound desc = could not find container \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": container with ID starting with 74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7 not found: ID does not exist" Jul 07 14:59:43.346610 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.346553 2566 scope.go:117] "RemoveContainer" containerID="f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" Jul 07 14:59:43.346780 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.346761 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974"} err="failed to get container status \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": rpc error: code = NotFound desc = could not find container \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": container with ID starting with f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974 not found: ID does not exist" Jul 07 14:59:43.346865 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.346782 2566 scope.go:117] "RemoveContainer" containerID="52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" Jul 07 14:59:43.347029 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347008 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724"} err="failed to get container status \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": rpc error: code = NotFound desc = could not find container \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": container with ID starting with 52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724 not found: ID does not exist" Jul 07 14:59:43.347106 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347031 2566 scope.go:117] "RemoveContainer" containerID="1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3" Jul 07 14:59:43.347307 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347293 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:59:43.347360 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347341 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3"} err="failed to get container status \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": rpc error: code = NotFound desc = could not find container \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": container with ID starting with 1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3 not found: ID does not exist" Jul 07 14:59:43.347399 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347364 2566 scope.go:117] "RemoveContainer" containerID="83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03" Jul 07 14:59:43.347611 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347592 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03"} err="failed to get container status \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": rpc error: code = NotFound desc = could not find container \"83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03\": container with ID starting with 83a722c8f5d8330aa73b387a280d1071eed07df6f88f2af3d732e7d6c7eecf03 not found: ID does not exist" Jul 07 14:59:43.347611 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347605 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbff8bbe-6289-484e-a99c-38c0af9ff6a9" containerName="console" Jul 07 14:59:43.347611 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347611 2566 scope.go:117] "RemoveContainer" containerID="7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347618 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbff8bbe-6289-484e-a99c-38c0af9ff6a9" containerName="console" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347629 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347634 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347642 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee76827a-c9a7-4851-9e41-a268db8c3043" containerName="console" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347650 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee76827a-c9a7-4851-9e41-a268db8c3043" containerName="console" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347658 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy-thanos" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347671 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy-thanos" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347681 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02236daa-01c6-4fcf-b497-444d8258a20b" containerName="registry" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347687 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="02236daa-01c6-4fcf-b497-444d8258a20b" containerName="registry" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347697 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="init-config-reloader" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347705 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="init-config-reloader" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347715 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="config-reloader" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347720 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="config-reloader" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347725 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="thanos-sidecar" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347730 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="thanos-sidecar" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347735 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy-web" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347740 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy-web" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347775 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="prometheus" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347785 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="prometheus" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347826 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3"} err="failed to get container status \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": rpc error: code = NotFound desc = could not find container \"7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3\": container with ID starting with 7354d8f83163a33dda836df981f36c4c09e4c86a3d21d158de3efe9f862c25e3 not found: ID does not exist" Jul 07 14:59:43.347829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347837 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="config-reloader" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347841 2566 scope.go:117] "RemoveContainer" containerID="82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347846 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbff8bbe-6289-484e-a99c-38c0af9ff6a9" containerName="console" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347853 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="thanos-sidecar" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347860 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy-web" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347868 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="prometheus" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347873 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347879 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee76827a-c9a7-4851-9e41-a268db8c3043" containerName="console" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347884 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" containerName="kube-rbac-proxy-thanos" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.347890 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="02236daa-01c6-4fcf-b497-444d8258a20b" containerName="registry" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.348030 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda"} err="failed to get container status \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": rpc error: code = NotFound desc = could not find container \"82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda\": container with ID starting with 82f13eb8d95021ec2e4ff86e0c281c7dc293522c022955509980c6624f0b4eda not found: ID does not exist" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.348131 2566 scope.go:117] "RemoveContainer" containerID="74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.348358 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7"} err="failed to get container status \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": rpc error: code = NotFound desc = could not find container \"74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7\": container with ID starting with 74f5eedfa86351b0c374b2ded4dfafe528f2c56d74719c2e21a72561128ea8a7 not found: ID does not exist" Jul 07 14:59:43.348587 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.348372 2566 scope.go:117] "RemoveContainer" containerID="f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974" Jul 07 14:59:43.349097 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.348597 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974"} err="failed to get container status \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": rpc error: code = NotFound desc = could not find container \"f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974\": container with ID starting with f384310c6d21b88f49bf463172fa43c08899ec42cd24523ef33b8477f8a9c974 not found: ID does not exist" Jul 07 14:59:43.349097 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.348624 2566 scope.go:117] "RemoveContainer" containerID="52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724" Jul 07 14:59:43.349097 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.348872 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724"} err="failed to get container status \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": rpc error: code = NotFound desc = could not find container \"52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724\": container with ID starting with 52b58c5dc04477e5c0a7fb13bb605bf3ad79fea963ea2b9ac9a7fb2254ff3724 not found: ID does not exist" Jul 07 14:59:43.349097 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.348893 2566 scope.go:117] "RemoveContainer" containerID="1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3" Jul 07 14:59:43.349224 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.349108 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3"} err="failed to get container status \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": rpc error: code = NotFound desc = could not find container \"1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3\": container with ID starting with 1a3e80536ef2727a6f63b3713293fce518fd588441ba2d4ae29bed3d669ef6d3 not found: ID does not exist" Jul 07 14:59:43.352235 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.352214 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.358202 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.358184 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 14:59:43.358357 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.358200 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 14:59:43.358357 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.358214 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 14:59:43.358866 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.358847 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 14:59:43.359007 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.358992 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 14:59:43.359230 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.359212 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7d70iuc4nbm89\"" Jul 07 14:59:43.359506 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.359488 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 14:59:43.359581 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.359524 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 14:59:43.359698 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.359679 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 14:59:43.359777 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.359731 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 14:59:43.359824 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.359791 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 14:59:43.359907 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.359894 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 14:59:43.360422 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.360408 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 14:59:43.361193 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.361157 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d7zq5\"" Jul 07 14:59:43.361304 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.361241 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 14:59:43.364227 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.364208 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 14:59:43.364996 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.364981 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 14:59:43.372178 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.372157 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:59:43.381442 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381410 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381557 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381453 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381557 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381483 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ab7024d-dd8d-441c-860c-900a27b8aa53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381674 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381555 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381674 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381601 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381674 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381632 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381674 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381660 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6t7w\" (UniqueName: \"kubernetes.io/projected/4ab7024d-dd8d-441c-860c-900a27b8aa53-kube-api-access-s6t7w\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381866 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381692 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381866 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381721 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381866 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381752 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381866 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381778 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-web-config\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381866 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381837 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.381866 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381863 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.382292 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381893 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.382292 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381920 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.382292 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381944 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.382292 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381971 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-config\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.382292 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.381993 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.382292 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.382082 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.382292 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.382170 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ab7024d-dd8d-441c-860c-900a27b8aa53-config-out\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483282 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483282 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483531 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483302 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483531 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483320 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483531 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483341 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483531 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483385 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-config\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483531 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483531 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483531 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483456 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ab7024d-dd8d-441c-860c-900a27b8aa53-config-out\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483531 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483490 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483531 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483519 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483934 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ab7024d-dd8d-441c-860c-900a27b8aa53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483934 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483571 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483934 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483934 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483934 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483653 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6t7w\" (UniqueName: \"kubernetes.io/projected/4ab7024d-dd8d-441c-860c-900a27b8aa53-kube-api-access-s6t7w\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483934 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483934 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483721 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483934 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483759 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.483934 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.483789 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-web-config\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.484411 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.484372 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.486457 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.484911 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.486457 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.484980 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.486457 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.485263 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.486457 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.486233 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.486457 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.486338 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.486457 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.486401 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.486841 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.486702 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-web-config\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.486841 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.486803 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.487444 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.487032 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.487444 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.487408 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.487636 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.487609 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4ab7024d-dd8d-441c-860c-900a27b8aa53-config-out\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.487636 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.487623 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4ab7024d-dd8d-441c-860c-900a27b8aa53-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.488152 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.488128 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-config\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.488829 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.488805 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4ab7024d-dd8d-441c-860c-900a27b8aa53-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.489143 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.489120 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.489398 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.489379 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.489470 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.489399 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.489608 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.489590 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4ab7024d-dd8d-441c-860c-900a27b8aa53-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.494403 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.494386 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6t7w\" (UniqueName: \"kubernetes.io/projected/4ab7024d-dd8d-441c-860c-900a27b8aa53-kube-api-access-s6t7w\") pod \"prometheus-k8s-0\" (UID: \"4ab7024d-dd8d-441c-860c-900a27b8aa53\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.661679 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.661589 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:59:43.808615 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:43.808542 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:59:43.810918 ip-10-0-130-13 kubenswrapper[2566]: W0707 14:59:43.810884 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab7024d_dd8d_441c_860c_900a27b8aa53.slice/crio-4dcd6546960d1ff18ebc678e4827ee30cc58a8dd3ab73a4a1eb7fe7329711a36 WatchSource:0}: Error finding container 4dcd6546960d1ff18ebc678e4827ee30cc58a8dd3ab73a4a1eb7fe7329711a36: Status 404 returned error can't find the container with id 4dcd6546960d1ff18ebc678e4827ee30cc58a8dd3ab73a4a1eb7fe7329711a36 Jul 07 14:59:44.296241 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:44.296202 2566 generic.go:356] "Generic (PLEG): container finished" podID="4ab7024d-dd8d-441c-860c-900a27b8aa53" containerID="f421236202e0947b7113af29f903a4202444d55339975c93c1d34e6821049ffd" exitCode=0 Jul 07 14:59:44.296415 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:44.296286 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ab7024d-dd8d-441c-860c-900a27b8aa53","Type":"ContainerDied","Data":"f421236202e0947b7113af29f903a4202444d55339975c93c1d34e6821049ffd"} Jul 07 14:59:44.296415 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:44.296355 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ab7024d-dd8d-441c-860c-900a27b8aa53","Type":"ContainerStarted","Data":"4dcd6546960d1ff18ebc678e4827ee30cc58a8dd3ab73a4a1eb7fe7329711a36"} Jul 07 14:59:45.310643 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:45.310612 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5" path="/var/lib/kubelet/pods/b71f511a-aa5e-4ddb-9fb4-7dc425d85ba5/volumes" Jul 07 14:59:45.311156 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:45.311137 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ab7024d-dd8d-441c-860c-900a27b8aa53","Type":"ContainerStarted","Data":"6b6cc0b3e96742638e00ebb07eb7e3d7d9479c23253475a4d1910446128ac254"} Jul 07 14:59:45.311216 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:45.311199 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ab7024d-dd8d-441c-860c-900a27b8aa53","Type":"ContainerStarted","Data":"ca7d3d90342cdb0c892e139255417dc9a363f49fa8afaa4cd44e64271518b9e4"} Jul 07 14:59:45.311298 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:45.311244 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ab7024d-dd8d-441c-860c-900a27b8aa53","Type":"ContainerStarted","Data":"a9d6c53f568c0b53df257ca14b3cc3a4a840b9c0475b455fd1199ba3a6fbf815"} Jul 07 14:59:45.311298 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:45.311273 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ab7024d-dd8d-441c-860c-900a27b8aa53","Type":"ContainerStarted","Data":"3632fd4dfb6c7bf236865ae418635edced84f04c499ff2a668606f3d8aae182d"} Jul 07 14:59:45.311298 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:45.311295 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ab7024d-dd8d-441c-860c-900a27b8aa53","Type":"ContainerStarted","Data":"39450deec32e104649d5d6a1465f74fb31f5deda196a5d12cca22b7dba19f2e0"} Jul 07 14:59:45.311387 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:45.311333 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4ab7024d-dd8d-441c-860c-900a27b8aa53","Type":"ContainerStarted","Data":"403c97cf660fefa8cbe284755ef56f15fe3ec68aa2c34b119952c2511111c67e"} Jul 07 14:59:45.341468 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:45.341409 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.3227734 podStartE2EDuration="2.341390174s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.616774ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:59:45.33900788 +0000 UTC m=+192.547065952" watchObservedRunningTime="2026-07-07 14:59:45.341390174 +0000 UTC m=+192.549448206" Jul 07 14:59:48.662211 ip-10-0-130-13 kubenswrapper[2566]: I0707 14:59:48.662159 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:00:43.661856 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:00:43.661824 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:00:43.677135 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:00:43.677104 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:00:44.539715 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:00:44.539688 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 15:01:33.175732 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:01:33.175700 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:01:33.177686 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:01:33.177658 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:01:33.186687 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:01:33.186665 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 15:04:07.411888 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.411853 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-wbkwl"] Jul 07 15:04:07.414989 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.414969 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:07.417827 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.417807 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jul 07 15:04:07.417827 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.417819 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 07 15:04:07.418017 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.417830 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-smfgn\"" Jul 07 15:04:07.418936 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.418917 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 07 15:04:07.428536 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.428513 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wbkwl"] Jul 07 15:04:07.516234 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.516199 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/211ee1ce-2f3d-4d72-820f-58ff699a3624-tls-certs\") pod \"model-serving-api-fd65d7d6-wbkwl\" (UID: \"211ee1ce-2f3d-4d72-820f-58ff699a3624\") " pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:07.516401 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.516241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6qd\" (UniqueName: \"kubernetes.io/projected/211ee1ce-2f3d-4d72-820f-58ff699a3624-kube-api-access-wh6qd\") pod \"model-serving-api-fd65d7d6-wbkwl\" (UID: \"211ee1ce-2f3d-4d72-820f-58ff699a3624\") " pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:07.616783 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.616749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/211ee1ce-2f3d-4d72-820f-58ff699a3624-tls-certs\") pod \"model-serving-api-fd65d7d6-wbkwl\" (UID: \"211ee1ce-2f3d-4d72-820f-58ff699a3624\") " pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:07.616939 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.616791 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6qd\" (UniqueName: \"kubernetes.io/projected/211ee1ce-2f3d-4d72-820f-58ff699a3624-kube-api-access-wh6qd\") pod \"model-serving-api-fd65d7d6-wbkwl\" (UID: \"211ee1ce-2f3d-4d72-820f-58ff699a3624\") " pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:07.619212 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.619179 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/211ee1ce-2f3d-4d72-820f-58ff699a3624-tls-certs\") pod \"model-serving-api-fd65d7d6-wbkwl\" (UID: \"211ee1ce-2f3d-4d72-820f-58ff699a3624\") " pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:07.627926 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.627900 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6qd\" (UniqueName: \"kubernetes.io/projected/211ee1ce-2f3d-4d72-820f-58ff699a3624-kube-api-access-wh6qd\") pod \"model-serving-api-fd65d7d6-wbkwl\" (UID: \"211ee1ce-2f3d-4d72-820f-58ff699a3624\") " pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:07.725315 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.725234 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:07.859603 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.859577 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wbkwl"] Jul 07 15:04:07.862000 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:04:07.861964 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211ee1ce_2f3d_4d72_820f_58ff699a3624.slice/crio-e4f0d995b821c732a4e11a468a1b7cf1acf736b71b18cb7c136c569c9bf78e67 WatchSource:0}: Error finding container e4f0d995b821c732a4e11a468a1b7cf1acf736b71b18cb7c136c569c9bf78e67: Status 404 returned error can't find the container with id e4f0d995b821c732a4e11a468a1b7cf1acf736b71b18cb7c136c569c9bf78e67 Jul 07 15:04:07.863780 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:07.863763 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:04:08.288269 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:08.288222 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wbkwl" event={"ID":"211ee1ce-2f3d-4d72-820f-58ff699a3624","Type":"ContainerStarted","Data":"e4f0d995b821c732a4e11a468a1b7cf1acf736b71b18cb7c136c569c9bf78e67"} Jul 07 15:04:10.298469 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:10.298375 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wbkwl" event={"ID":"211ee1ce-2f3d-4d72-820f-58ff699a3624","Type":"ContainerStarted","Data":"6d23832117bc94e28d84bf4fa7b0b8946b40cbf5f19d026a0f3a2d2de8926a19"} Jul 07 15:04:10.298822 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:10.298491 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:10.318525 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:10.318479 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-wbkwl" podStartSLOduration=1.164561211 podStartE2EDuration="3.318464843s" totalImagesPullingTime="2.153903632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:04:10.317066668 +0000 UTC m=+457.525124691" watchObservedRunningTime="2026-07-07 15:04:10.318464843 +0000 UTC m=+457.526522875" Jul 07 15:04:21.310500 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:21.310471 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-wbkwl" Jul 07 15:04:23.372664 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.372629 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9twvw"] Jul 07 15:04:23.377694 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.377675 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9twvw" Jul 07 15:04:23.380421 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.380396 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 07 15:04:23.380535 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.380483 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jvqqn\"" Jul 07 15:04:23.384951 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.384926 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9twvw"] Jul 07 15:04:23.551158 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.551118 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z922c\" (UniqueName: \"kubernetes.io/projected/47fe24a4-1beb-4cbd-8423-475fba20ebe2-kube-api-access-z922c\") pod \"s3-init-9twvw\" (UID: \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\") " pod="kserve/s3-init-9twvw" Jul 07 15:04:23.551158 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.551161 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fe24a4-1beb-4cbd-8423-475fba20ebe2-model-cache\") pod \"s3-init-9twvw\" (UID: \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\") " pod="kserve/s3-init-9twvw" Jul 07 15:04:23.652296 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.652187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fe24a4-1beb-4cbd-8423-475fba20ebe2-model-cache\") pod \"s3-init-9twvw\" (UID: \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\") " pod="kserve/s3-init-9twvw" Jul 07 15:04:23.652457 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.652305 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z922c\" (UniqueName: \"kubernetes.io/projected/47fe24a4-1beb-4cbd-8423-475fba20ebe2-kube-api-access-z922c\") pod \"s3-init-9twvw\" (UID: \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\") " pod="kserve/s3-init-9twvw" Jul 07 15:04:23.652637 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.652611 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fe24a4-1beb-4cbd-8423-475fba20ebe2-model-cache\") pod \"s3-init-9twvw\" (UID: \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\") " pod="kserve/s3-init-9twvw" Jul 07 15:04:23.662816 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.662793 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z922c\" (UniqueName: \"kubernetes.io/projected/47fe24a4-1beb-4cbd-8423-475fba20ebe2-kube-api-access-z922c\") pod \"s3-init-9twvw\" (UID: \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\") " pod="kserve/s3-init-9twvw" Jul 07 15:04:23.687915 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.687887 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9twvw" Jul 07 15:04:23.821385 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:23.821350 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9twvw"] Jul 07 15:04:23.825177 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:04:23.825150 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47fe24a4_1beb_4cbd_8423_475fba20ebe2.slice/crio-1e1c04b3fd9793512cddb110a90118360a102a891b21447088aeead5438ec3b2 WatchSource:0}: Error finding container 1e1c04b3fd9793512cddb110a90118360a102a891b21447088aeead5438ec3b2: Status 404 returned error can't find the container with id 1e1c04b3fd9793512cddb110a90118360a102a891b21447088aeead5438ec3b2 Jul 07 15:04:24.352313 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:24.352278 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9twvw" event={"ID":"47fe24a4-1beb-4cbd-8423-475fba20ebe2","Type":"ContainerStarted","Data":"1e1c04b3fd9793512cddb110a90118360a102a891b21447088aeead5438ec3b2"} Jul 07 15:04:28.370238 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:28.370200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9twvw" event={"ID":"47fe24a4-1beb-4cbd-8423-475fba20ebe2","Type":"ContainerStarted","Data":"46097806915ff9727d0e49b4ef402f19421524ab84aca1b133327be44d7c00fb"} Jul 07 15:04:30.380611 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:30.380575 2566 generic.go:356] "Generic (PLEG): container finished" podID="47fe24a4-1beb-4cbd-8423-475fba20ebe2" containerID="46097806915ff9727d0e49b4ef402f19421524ab84aca1b133327be44d7c00fb" exitCode=0 Jul 07 15:04:30.380981 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:30.380655 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9twvw" event={"ID":"47fe24a4-1beb-4cbd-8423-475fba20ebe2","Type":"ContainerDied","Data":"46097806915ff9727d0e49b4ef402f19421524ab84aca1b133327be44d7c00fb"} Jul 07 15:04:33.212347 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:33.212307 2566 scope.go:117] "RemoveContainer" containerID="16369ad04d1c570fb40ebaa867e94e40153774919540c33baa6d1387c32c381f" Jul 07 15:04:34.953910 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:34.953882 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 07 15:04:35.406988 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:35.406953 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9twvw" event={"ID":"47fe24a4-1beb-4cbd-8423-475fba20ebe2","Type":"ContainerStarted","Data":"df3410bdf785f7ac3f29bb2bffd373cbb1988ac010f341c66d6952819c4517e7"} Jul 07 15:04:35.440992 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:04:35.440940 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9twvw" podStartSLOduration=2.272889661 podStartE2EDuration="12.440924702s" totalImagesPullingTime="8.575481719s" totalInitContainerRuntime="1.592553322s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:04:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:04:35.439861444 +0000 UTC m=+482.647919488" watchObservedRunningTime="2026-07-07 15:04:35.440924702 +0000 UTC m=+482.648982734" Jul 07 15:05:40.654604 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:40.654571 2566 generic.go:356] "Generic (PLEG): container finished" podID="47fe24a4-1beb-4cbd-8423-475fba20ebe2" containerID="df3410bdf785f7ac3f29bb2bffd373cbb1988ac010f341c66d6952819c4517e7" exitCode=0 Jul 07 15:05:40.655026 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:40.654644 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9twvw" event={"ID":"47fe24a4-1beb-4cbd-8423-475fba20ebe2","Type":"ContainerDied","Data":"df3410bdf785f7ac3f29bb2bffd373cbb1988ac010f341c66d6952819c4517e7"} Jul 07 15:05:41.777884 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:41.777857 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9twvw" Jul 07 15:05:41.788197 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:41.788176 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z922c\" (UniqueName: \"kubernetes.io/projected/47fe24a4-1beb-4cbd-8423-475fba20ebe2-kube-api-access-z922c\") pod \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\" (UID: \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\") " Jul 07 15:05:41.788313 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:41.788220 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fe24a4-1beb-4cbd-8423-475fba20ebe2-model-cache\") pod \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\" (UID: \"47fe24a4-1beb-4cbd-8423-475fba20ebe2\") " Jul 07 15:05:41.795719 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:41.795685 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fe24a4-1beb-4cbd-8423-475fba20ebe2-kube-api-access-z922c" (OuterVolumeSpecName: "kube-api-access-z922c") pod "47fe24a4-1beb-4cbd-8423-475fba20ebe2" (UID: "47fe24a4-1beb-4cbd-8423-475fba20ebe2"). InnerVolumeSpecName "kube-api-access-z922c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:05:41.813013 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:41.812984 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fe24a4-1beb-4cbd-8423-475fba20ebe2-model-cache" (OuterVolumeSpecName: "model-cache") pod "47fe24a4-1beb-4cbd-8423-475fba20ebe2" (UID: "47fe24a4-1beb-4cbd-8423-475fba20ebe2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:41.888908 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:41.888877 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47fe24a4-1beb-4cbd-8423-475fba20ebe2-model-cache\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:05:41.888908 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:41.888910 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z922c\" (UniqueName: \"kubernetes.io/projected/47fe24a4-1beb-4cbd-8423-475fba20ebe2-kube-api-access-z922c\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:05:42.663553 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:42.663525 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9twvw" Jul 07 15:05:42.663711 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:42.663525 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9twvw" event={"ID":"47fe24a4-1beb-4cbd-8423-475fba20ebe2","Type":"ContainerDied","Data":"1e1c04b3fd9793512cddb110a90118360a102a891b21447088aeead5438ec3b2"} Jul 07 15:05:42.663711 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:42.663634 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e1c04b3fd9793512cddb110a90118360a102a891b21447088aeead5438ec3b2" Jul 07 15:05:53.286849 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.286815 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-7gktl"] Jul 07 15:05:53.287272 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.287130 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fe24a4-1beb-4cbd-8423-475fba20ebe2" containerName="download-hf-model" Jul 07 15:05:53.287272 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.287143 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fe24a4-1beb-4cbd-8423-475fba20ebe2" containerName="download-hf-model" Jul 07 15:05:53.287272 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.287151 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fe24a4-1beb-4cbd-8423-475fba20ebe2" containerName="s3-init" Jul 07 15:05:53.287272 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.287157 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fe24a4-1beb-4cbd-8423-475fba20ebe2" containerName="s3-init" Jul 07 15:05:53.287272 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.287207 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="47fe24a4-1beb-4cbd-8423-475fba20ebe2" containerName="s3-init" Jul 07 15:05:53.289268 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.289250 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:05:53.292645 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.292620 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\"" Jul 07 15:05:53.292778 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.292643 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jvqqn\"" Jul 07 15:05:53.304874 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.304853 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7gktl"] Jul 07 15:05:53.369289 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.369267 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2132551-2e35-43c7-9c0e-793ed23b2920-model-cache\") pod \"s3-tls-init-custom-7gktl\" (UID: \"f2132551-2e35-43c7-9c0e-793ed23b2920\") " pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:05:53.369417 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.369309 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtz79\" (UniqueName: \"kubernetes.io/projected/f2132551-2e35-43c7-9c0e-793ed23b2920-kube-api-access-gtz79\") pod \"s3-tls-init-custom-7gktl\" (UID: \"f2132551-2e35-43c7-9c0e-793ed23b2920\") " pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:05:53.470452 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.470421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2132551-2e35-43c7-9c0e-793ed23b2920-model-cache\") pod \"s3-tls-init-custom-7gktl\" (UID: \"f2132551-2e35-43c7-9c0e-793ed23b2920\") " pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:05:53.470578 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.470463 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtz79\" (UniqueName: \"kubernetes.io/projected/f2132551-2e35-43c7-9c0e-793ed23b2920-kube-api-access-gtz79\") pod \"s3-tls-init-custom-7gktl\" (UID: \"f2132551-2e35-43c7-9c0e-793ed23b2920\") " pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:05:53.470801 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.470772 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2132551-2e35-43c7-9c0e-793ed23b2920-model-cache\") pod \"s3-tls-init-custom-7gktl\" (UID: \"f2132551-2e35-43c7-9c0e-793ed23b2920\") " pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:05:53.483237 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.483209 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtz79\" (UniqueName: \"kubernetes.io/projected/f2132551-2e35-43c7-9c0e-793ed23b2920-kube-api-access-gtz79\") pod \"s3-tls-init-custom-7gktl\" (UID: \"f2132551-2e35-43c7-9c0e-793ed23b2920\") " pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:05:53.598515 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.598449 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:05:53.734924 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:53.734848 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7gktl"] Jul 07 15:05:53.737019 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:05:53.736985 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2132551_2e35_43c7_9c0e_793ed23b2920.slice/crio-75704e98499c290120b76719cc1d0335336584e738b802f8df261b9801f1023a WatchSource:0}: Error finding container 75704e98499c290120b76719cc1d0335336584e738b802f8df261b9801f1023a: Status 404 returned error can't find the container with id 75704e98499c290120b76719cc1d0335336584e738b802f8df261b9801f1023a Jul 07 15:05:54.710835 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:54.710791 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7gktl" event={"ID":"f2132551-2e35-43c7-9c0e-793ed23b2920","Type":"ContainerStarted","Data":"46057b8ac6f946dc45ca172161c63165c0c5aec850fd030f00a203241ca18f00"} Jul 07 15:05:54.710835 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:54.710838 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7gktl" event={"ID":"f2132551-2e35-43c7-9c0e-793ed23b2920","Type":"ContainerStarted","Data":"75704e98499c290120b76719cc1d0335336584e738b802f8df261b9801f1023a"} Jul 07 15:05:55.716242 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:55.716204 2566 generic.go:356] "Generic (PLEG): container finished" podID="f2132551-2e35-43c7-9c0e-793ed23b2920" containerID="46057b8ac6f946dc45ca172161c63165c0c5aec850fd030f00a203241ca18f00" exitCode=0 Jul 07 15:05:55.716649 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:55.716301 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7gktl" event={"ID":"f2132551-2e35-43c7-9c0e-793ed23b2920","Type":"ContainerDied","Data":"46057b8ac6f946dc45ca172161c63165c0c5aec850fd030f00a203241ca18f00"} Jul 07 15:05:56.721973 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:05:56.721932 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7gktl" event={"ID":"f2132551-2e35-43c7-9c0e-793ed23b2920","Type":"ContainerStarted","Data":"b93cf153a60a611edbe7b2f131d78bde178b5546207b6436583a94da043f5dcf"} Jul 07 15:06:01.744040 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:01.744009 2566 generic.go:356] "Generic (PLEG): container finished" podID="f2132551-2e35-43c7-9c0e-793ed23b2920" containerID="b93cf153a60a611edbe7b2f131d78bde178b5546207b6436583a94da043f5dcf" exitCode=0 Jul 07 15:06:01.744396 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:01.744083 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7gktl" event={"ID":"f2132551-2e35-43c7-9c0e-793ed23b2920","Type":"ContainerDied","Data":"b93cf153a60a611edbe7b2f131d78bde178b5546207b6436583a94da043f5dcf"} Jul 07 15:06:02.885970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:02.885938 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:06:02.948802 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:02.948774 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2132551-2e35-43c7-9c0e-793ed23b2920-model-cache\") pod \"f2132551-2e35-43c7-9c0e-793ed23b2920\" (UID: \"f2132551-2e35-43c7-9c0e-793ed23b2920\") " Jul 07 15:06:02.948932 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:02.948828 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtz79\" (UniqueName: \"kubernetes.io/projected/f2132551-2e35-43c7-9c0e-793ed23b2920-kube-api-access-gtz79\") pod \"f2132551-2e35-43c7-9c0e-793ed23b2920\" (UID: \"f2132551-2e35-43c7-9c0e-793ed23b2920\") " Jul 07 15:06:02.950832 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:02.950794 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2132551-2e35-43c7-9c0e-793ed23b2920-kube-api-access-gtz79" (OuterVolumeSpecName: "kube-api-access-gtz79") pod "f2132551-2e35-43c7-9c0e-793ed23b2920" (UID: "f2132551-2e35-43c7-9c0e-793ed23b2920"). InnerVolumeSpecName "kube-api-access-gtz79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:06:02.970741 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:02.970716 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2132551-2e35-43c7-9c0e-793ed23b2920-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2132551-2e35-43c7-9c0e-793ed23b2920" (UID: "f2132551-2e35-43c7-9c0e-793ed23b2920"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:06:03.049451 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:03.049428 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtz79\" (UniqueName: \"kubernetes.io/projected/f2132551-2e35-43c7-9c0e-793ed23b2920-kube-api-access-gtz79\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:06:03.049451 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:03.049449 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2132551-2e35-43c7-9c0e-793ed23b2920-model-cache\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:06:03.752850 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:03.752821 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7gktl" Jul 07 15:06:03.753006 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:03.752812 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7gktl" event={"ID":"f2132551-2e35-43c7-9c0e-793ed23b2920","Type":"ContainerDied","Data":"75704e98499c290120b76719cc1d0335336584e738b802f8df261b9801f1023a"} Jul 07 15:06:03.753006 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:03.752945 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75704e98499c290120b76719cc1d0335336584e738b802f8df261b9801f1023a" Jul 07 15:06:07.281836 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.281792 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-9zsf7"] Jul 07 15:06:07.282362 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.282258 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2132551-2e35-43c7-9c0e-793ed23b2920" containerName="download-hf-model" Jul 07 15:06:07.282362 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.282277 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2132551-2e35-43c7-9c0e-793ed23b2920" containerName="download-hf-model" Jul 07 15:06:07.282362 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.282308 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2132551-2e35-43c7-9c0e-793ed23b2920" containerName="s3-tls-init-custom" Jul 07 15:06:07.282362 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.282315 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2132551-2e35-43c7-9c0e-793ed23b2920" containerName="s3-tls-init-custom" Jul 07 15:06:07.282577 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.282390 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2132551-2e35-43c7-9c0e-793ed23b2920" containerName="s3-tls-init-custom" Jul 07 15:06:07.284454 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.284434 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:07.287150 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.287121 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jvqqn\"" Jul 07 15:06:07.287372 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.287354 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\"" Jul 07 15:06:07.310938 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.310915 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9zsf7"] Jul 07 15:06:07.385059 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.385011 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db8ebce-37ce-4e5e-b012-f2da9772befb-model-cache\") pod \"s3-tls-init-serving-9zsf7\" (UID: \"2db8ebce-37ce-4e5e-b012-f2da9772befb\") " pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:07.385224 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.385069 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcb2r\" (UniqueName: \"kubernetes.io/projected/2db8ebce-37ce-4e5e-b012-f2da9772befb-kube-api-access-qcb2r\") pod \"s3-tls-init-serving-9zsf7\" (UID: \"2db8ebce-37ce-4e5e-b012-f2da9772befb\") " pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:07.486303 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.486252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcb2r\" (UniqueName: \"kubernetes.io/projected/2db8ebce-37ce-4e5e-b012-f2da9772befb-kube-api-access-qcb2r\") pod \"s3-tls-init-serving-9zsf7\" (UID: \"2db8ebce-37ce-4e5e-b012-f2da9772befb\") " pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:07.486472 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.486338 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db8ebce-37ce-4e5e-b012-f2da9772befb-model-cache\") pod \"s3-tls-init-serving-9zsf7\" (UID: \"2db8ebce-37ce-4e5e-b012-f2da9772befb\") " pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:07.486714 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.486698 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db8ebce-37ce-4e5e-b012-f2da9772befb-model-cache\") pod \"s3-tls-init-serving-9zsf7\" (UID: \"2db8ebce-37ce-4e5e-b012-f2da9772befb\") " pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:07.496874 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.496842 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcb2r\" (UniqueName: \"kubernetes.io/projected/2db8ebce-37ce-4e5e-b012-f2da9772befb-kube-api-access-qcb2r\") pod \"s3-tls-init-serving-9zsf7\" (UID: \"2db8ebce-37ce-4e5e-b012-f2da9772befb\") " pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:07.593427 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.593356 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:07.735214 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.735191 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9zsf7"] Jul 07 15:06:07.737862 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:06:07.737836 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2db8ebce_37ce_4e5e_b012_f2da9772befb.slice/crio-49f50bbc13e9291c93d6b4598f6db1052cc4825434fd9001f1c0143a8ba840cf WatchSource:0}: Error finding container 49f50bbc13e9291c93d6b4598f6db1052cc4825434fd9001f1c0143a8ba840cf: Status 404 returned error can't find the container with id 49f50bbc13e9291c93d6b4598f6db1052cc4825434fd9001f1c0143a8ba840cf Jul 07 15:06:07.769187 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:07.769160 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9zsf7" event={"ID":"2db8ebce-37ce-4e5e-b012-f2da9772befb","Type":"ContainerStarted","Data":"49f50bbc13e9291c93d6b4598f6db1052cc4825434fd9001f1c0143a8ba840cf"} Jul 07 15:06:08.775600 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:08.775543 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9zsf7" event={"ID":"2db8ebce-37ce-4e5e-b012-f2da9772befb","Type":"ContainerStarted","Data":"47340ee9da4cf3b52cc217264363b4124112c9ef56161fad95efbeda72b1212b"} Jul 07 15:06:09.781461 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:09.781427 2566 generic.go:356] "Generic (PLEG): container finished" podID="2db8ebce-37ce-4e5e-b012-f2da9772befb" containerID="47340ee9da4cf3b52cc217264363b4124112c9ef56161fad95efbeda72b1212b" exitCode=0 Jul 07 15:06:09.781903 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:09.781517 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9zsf7" event={"ID":"2db8ebce-37ce-4e5e-b012-f2da9772befb","Type":"ContainerDied","Data":"47340ee9da4cf3b52cc217264363b4124112c9ef56161fad95efbeda72b1212b"} Jul 07 15:06:10.787614 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:10.787574 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9zsf7" event={"ID":"2db8ebce-37ce-4e5e-b012-f2da9772befb","Type":"ContainerStarted","Data":"bd4e30b2e1d6e4659d97cba92a54a864417ce9e02de04fa67a01587e12fbee9e"} Jul 07 15:06:10.808597 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:10.808547 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-9zsf7" podStartSLOduration=2.264144361 podStartE2EDuration="3.808529811s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.54438545s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:06:10.80669386 +0000 UTC m=+578.014751891" watchObservedRunningTime="2026-07-07 15:06:10.808529811 +0000 UTC m=+578.016587843" Jul 07 15:06:16.815867 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:16.815832 2566 generic.go:356] "Generic (PLEG): container finished" podID="2db8ebce-37ce-4e5e-b012-f2da9772befb" containerID="bd4e30b2e1d6e4659d97cba92a54a864417ce9e02de04fa67a01587e12fbee9e" exitCode=0 Jul 07 15:06:16.816283 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:16.815904 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9zsf7" event={"ID":"2db8ebce-37ce-4e5e-b012-f2da9772befb","Type":"ContainerDied","Data":"bd4e30b2e1d6e4659d97cba92a54a864417ce9e02de04fa67a01587e12fbee9e"} Jul 07 15:06:17.954576 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:17.954554 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:18.077392 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:18.077314 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db8ebce-37ce-4e5e-b012-f2da9772befb-model-cache\") pod \"2db8ebce-37ce-4e5e-b012-f2da9772befb\" (UID: \"2db8ebce-37ce-4e5e-b012-f2da9772befb\") " Jul 07 15:06:18.077392 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:18.077380 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcb2r\" (UniqueName: \"kubernetes.io/projected/2db8ebce-37ce-4e5e-b012-f2da9772befb-kube-api-access-qcb2r\") pod \"2db8ebce-37ce-4e5e-b012-f2da9772befb\" (UID: \"2db8ebce-37ce-4e5e-b012-f2da9772befb\") " Jul 07 15:06:18.079524 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:18.079495 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db8ebce-37ce-4e5e-b012-f2da9772befb-kube-api-access-qcb2r" (OuterVolumeSpecName: "kube-api-access-qcb2r") pod "2db8ebce-37ce-4e5e-b012-f2da9772befb" (UID: "2db8ebce-37ce-4e5e-b012-f2da9772befb"). InnerVolumeSpecName "kube-api-access-qcb2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:06:18.100208 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:18.100163 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db8ebce-37ce-4e5e-b012-f2da9772befb-model-cache" (OuterVolumeSpecName: "model-cache") pod "2db8ebce-37ce-4e5e-b012-f2da9772befb" (UID: "2db8ebce-37ce-4e5e-b012-f2da9772befb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:06:18.177906 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:18.177884 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcb2r\" (UniqueName: \"kubernetes.io/projected/2db8ebce-37ce-4e5e-b012-f2da9772befb-kube-api-access-qcb2r\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:06:18.177906 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:18.177905 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2db8ebce-37ce-4e5e-b012-f2da9772befb-model-cache\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:06:18.825612 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:18.825583 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9zsf7" Jul 07 15:06:18.825612 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:18.825588 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9zsf7" event={"ID":"2db8ebce-37ce-4e5e-b012-f2da9772befb","Type":"ContainerDied","Data":"49f50bbc13e9291c93d6b4598f6db1052cc4825434fd9001f1c0143a8ba840cf"} Jul 07 15:06:18.825815 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:18.825629 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49f50bbc13e9291c93d6b4598f6db1052cc4825434fd9001f1c0143a8ba840cf" Jul 07 15:06:33.202900 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:33.202870 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:06:33.203372 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:06:33.203250 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:11:33.227731 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:33.227647 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:11:33.228786 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:33.228763 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:11:46.828272 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.828237 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k"] Jul 07 15:11:46.828649 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.828522 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db8ebce-37ce-4e5e-b012-f2da9772befb" containerName="s3-tls-init-serving" Jul 07 15:11:46.828649 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.828532 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db8ebce-37ce-4e5e-b012-f2da9772befb" containerName="s3-tls-init-serving" Jul 07 15:11:46.828649 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.828548 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db8ebce-37ce-4e5e-b012-f2da9772befb" containerName="download-hf-model" Jul 07 15:11:46.828649 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.828553 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db8ebce-37ce-4e5e-b012-f2da9772befb" containerName="download-hf-model" Jul 07 15:11:46.828649 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.828613 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db8ebce-37ce-4e5e-b012-f2da9772befb" containerName="s3-tls-init-serving" Jul 07 15:11:46.831651 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.831634 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:46.834447 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.834418 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\"" Jul 07 15:11:46.834586 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.834545 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\"" Jul 07 15:11:46.834586 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.834581 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 15:11:46.834801 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.834785 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 07 15:11:46.834866 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.834822 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 15:11:46.835752 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.835721 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-bnv6j\"" Jul 07 15:11:46.835835 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.835750 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 07 15:11:46.844891 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.844866 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k"] Jul 07 15:11:46.920620 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.920586 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f60202a-6a6c-45e2-82e3-ade8129035c8-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:46.920620 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.920634 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hzb\" (UniqueName: \"kubernetes.io/projected/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-api-access-64hzb\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:46.920841 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.920669 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:46.920841 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.920711 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:46.920841 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:46.920756 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f60202a-6a6c-45e2-82e3-ade8129035c8-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.021293 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.021263 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64hzb\" (UniqueName: \"kubernetes.io/projected/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-api-access-64hzb\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.021469 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.021309 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.021469 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.021348 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.021469 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.021379 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f60202a-6a6c-45e2-82e3-ade8129035c8-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.021469 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.021413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f60202a-6a6c-45e2-82e3-ade8129035c8-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.021883 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.021853 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f60202a-6a6c-45e2-82e3-ade8129035c8-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.022112 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.022096 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.022174 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.022109 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.023983 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.023955 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f60202a-6a6c-45e2-82e3-ade8129035c8-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.031641 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.031621 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hzb\" (UniqueName: \"kubernetes.io/projected/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-api-access-64hzb\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.144701 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.144618 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:11:47.278207 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:47.278183 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k"] Jul 07 15:11:47.280770 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:11:47.280727 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f60202a_6a6c_45e2_82e3_ade8129035c8.slice/crio-11670e7a1be6e9ea86895fa1a2bfb78812abcda06f25f8d39f54ba7c136d5566 WatchSource:0}: Error finding container 11670e7a1be6e9ea86895fa1a2bfb78812abcda06f25f8d39f54ba7c136d5566: Status 404 returned error can't find the container with id 11670e7a1be6e9ea86895fa1a2bfb78812abcda06f25f8d39f54ba7c136d5566 Jul 07 15:11:48.117970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:48.117935 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" event={"ID":"5f60202a-6a6c-45e2-82e3-ade8129035c8","Type":"ContainerStarted","Data":"392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11"} Jul 07 15:11:48.118391 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:11:48.117997 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" event={"ID":"5f60202a-6a6c-45e2-82e3-ade8129035c8","Type":"ContainerStarted","Data":"11670e7a1be6e9ea86895fa1a2bfb78812abcda06f25f8d39f54ba7c136d5566"} Jul 07 15:12:17.239032 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:12:17.238993 2566 generic.go:356] "Generic (PLEG): container finished" podID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerID="392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11" exitCode=0 Jul 07 15:12:17.239529 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:12:17.239069 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" event={"ID":"5f60202a-6a6c-45e2-82e3-ade8129035c8","Type":"ContainerDied","Data":"392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11"} Jul 07 15:12:17.240425 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:12:17.240406 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:14:45.906472 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:14:45.906419 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" event={"ID":"5f60202a-6a6c-45e2-82e3-ade8129035c8","Type":"ContainerStarted","Data":"f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248"} Jul 07 15:14:48.923386 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:14:48.923344 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" event={"ID":"5f60202a-6a6c-45e2-82e3-ade8129035c8","Type":"ContainerStarted","Data":"0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe"} Jul 07 15:14:48.923857 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:14:48.923470 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:14:48.955885 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:14:48.955827 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" podStartSLOduration=2.670444215 podStartE2EDuration="3m2.955809928s" totalImagesPullingTime="2m30.948465117s" totalInitContainerRuntime="29.336900596s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:11:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:14:48.955017544 +0000 UTC m=+1096.163075576" watchObservedRunningTime="2026-07-07 15:14:48.955809928 +0000 UTC m=+1096.163867959" Jul 07 15:14:49.927933 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:14:49.927897 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:14:49.929133 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:14:49.929104 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 07 15:14:50.932151 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:14:50.932110 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 07 15:14:55.938189 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:14:55.938156 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:14:55.938799 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:14:55.938736 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Jul 07 15:15:05.939900 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:05.939867 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:15:06.952562 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:06.952532 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k"] Jul 07 15:15:06.953074 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:06.952740 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kserve-container" containerID="cri-o://f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248" gracePeriod=30 Jul 07 15:15:06.953074 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:06.952770 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kube-rbac-proxy" containerID="cri-o://0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe" gracePeriod=30 Jul 07 15:15:07.825531 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.825501 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:15:07.928841 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.928744 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f60202a-6a6c-45e2-82e3-ade8129035c8-kserve-provision-location\") pod \"5f60202a-6a6c-45e2-82e3-ade8129035c8\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " Jul 07 15:15:07.928841 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.928811 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64hzb\" (UniqueName: \"kubernetes.io/projected/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-api-access-64hzb\") pod \"5f60202a-6a6c-45e2-82e3-ade8129035c8\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " Jul 07 15:15:07.929095 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.928850 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-cabundle-cert\") pod \"5f60202a-6a6c-45e2-82e3-ade8129035c8\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " Jul 07 15:15:07.929095 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.928888 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f60202a-6a6c-45e2-82e3-ade8129035c8-proxy-tls\") pod \"5f60202a-6a6c-45e2-82e3-ade8129035c8\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " Jul 07 15:15:07.929095 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.928908 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-rbac-proxy-sar-config\") pod \"5f60202a-6a6c-45e2-82e3-ade8129035c8\" (UID: \"5f60202a-6a6c-45e2-82e3-ade8129035c8\") " Jul 07 15:15:07.929304 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.929273 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f60202a-6a6c-45e2-82e3-ade8129035c8" (UID: "5f60202a-6a6c-45e2-82e3-ade8129035c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:15:07.929376 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.929355 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f60202a-6a6c-45e2-82e3-ade8129035c8" (UID: "5f60202a-6a6c-45e2-82e3-ade8129035c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:15:07.931282 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.931257 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-api-access-64hzb" (OuterVolumeSpecName: "kube-api-access-64hzb") pod "5f60202a-6a6c-45e2-82e3-ade8129035c8" (UID: "5f60202a-6a6c-45e2-82e3-ade8129035c8"). InnerVolumeSpecName "kube-api-access-64hzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:15:07.931392 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.931304 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f60202a-6a6c-45e2-82e3-ade8129035c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f60202a-6a6c-45e2-82e3-ade8129035c8" (UID: "5f60202a-6a6c-45e2-82e3-ade8129035c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:15:07.935346 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:07.935323 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f60202a-6a6c-45e2-82e3-ade8129035c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f60202a-6a6c-45e2-82e3-ade8129035c8" (UID: "5f60202a-6a6c-45e2-82e3-ade8129035c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:15:08.002921 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.002879 2566 generic.go:356] "Generic (PLEG): container finished" podID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerID="0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe" exitCode=2 Jul 07 15:15:08.002921 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.002915 2566 generic.go:356] "Generic (PLEG): container finished" podID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerID="f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248" exitCode=0 Jul 07 15:15:08.003406 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.002946 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" Jul 07 15:15:08.003406 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.002959 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" event={"ID":"5f60202a-6a6c-45e2-82e3-ade8129035c8","Type":"ContainerDied","Data":"0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe"} Jul 07 15:15:08.003406 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.003026 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" event={"ID":"5f60202a-6a6c-45e2-82e3-ade8129035c8","Type":"ContainerDied","Data":"f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248"} Jul 07 15:15:08.003406 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.003073 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k" event={"ID":"5f60202a-6a6c-45e2-82e3-ade8129035c8","Type":"ContainerDied","Data":"11670e7a1be6e9ea86895fa1a2bfb78812abcda06f25f8d39f54ba7c136d5566"} Jul 07 15:15:08.003406 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.003100 2566 scope.go:117] "RemoveContainer" containerID="0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe" Jul 07 15:15:08.011935 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.011918 2566 scope.go:117] "RemoveContainer" containerID="f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248" Jul 07 15:15:08.019180 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.019161 2566 scope.go:117] "RemoveContainer" containerID="392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11" Jul 07 15:15:08.026386 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.026364 2566 scope.go:117] "RemoveContainer" containerID="0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe" Jul 07 15:15:08.026660 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:15:08.026640 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe\": container with ID starting with 0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe not found: ID does not exist" containerID="0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe" Jul 07 15:15:08.026733 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.026667 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe"} err="failed to get container status \"0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe\": rpc error: code = NotFound desc = could not find container \"0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe\": container with ID starting with 0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe not found: ID does not exist" Jul 07 15:15:08.026733 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.026688 2566 scope.go:117] "RemoveContainer" containerID="f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248" Jul 07 15:15:08.026930 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:15:08.026906 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248\": container with ID starting with f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248 not found: ID does not exist" containerID="f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248" Jul 07 15:15:08.026985 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.026935 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248"} err="failed to get container status \"f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248\": rpc error: code = NotFound desc = could not find container \"f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248\": container with ID starting with f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248 not found: ID does not exist" Jul 07 15:15:08.026985 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.026953 2566 scope.go:117] "RemoveContainer" containerID="392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11" Jul 07 15:15:08.027118 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.027035 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k"] Jul 07 15:15:08.027188 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:15:08.027171 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11\": container with ID starting with 392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11 not found: ID does not exist" containerID="392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11" Jul 07 15:15:08.027236 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.027192 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11"} err="failed to get container status \"392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11\": rpc error: code = NotFound desc = could not find container \"392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11\": container with ID starting with 392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11 not found: ID does not exist" Jul 07 15:15:08.027236 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.027206 2566 scope.go:117] "RemoveContainer" containerID="0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe" Jul 07 15:15:08.027441 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.027420 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe"} err="failed to get container status \"0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe\": rpc error: code = NotFound desc = could not find container \"0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe\": container with ID starting with 0c0d2328db002291a016110c4bf61d80b70a1ae6a5694e73ed7a2d01c0a54fbe not found: ID does not exist" Jul 07 15:15:08.027514 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.027445 2566 scope.go:117] "RemoveContainer" containerID="f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248" Jul 07 15:15:08.027705 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.027686 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248"} err="failed to get container status \"f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248\": rpc error: code = NotFound desc = could not find container \"f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248\": container with ID starting with f090d0b7163a885308978667bb28cab562a8d98b7e0c1741e581aee33b318248 not found: ID does not exist" Jul 07 15:15:08.027755 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.027705 2566 scope.go:117] "RemoveContainer" containerID="392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11" Jul 07 15:15:08.027907 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.027891 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11"} err="failed to get container status \"392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11\": rpc error: code = NotFound desc = could not find container \"392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11\": container with ID starting with 392d5f3ddfc34bad861df00dee1ddb28f1e2d60a8c010dfc4fba251f2bd28e11 not found: ID does not exist" Jul 07 15:15:08.030284 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.030264 2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f60202a-6a6c-45e2-82e3-ade8129035c8-proxy-tls\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:15:08.030396 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.030286 2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:15:08.030396 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.030301 2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f60202a-6a6c-45e2-82e3-ade8129035c8-kserve-provision-location\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:15:08.030396 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.030317 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hzb\" (UniqueName: \"kubernetes.io/projected/5f60202a-6a6c-45e2-82e3-ade8129035c8-kube-api-access-64hzb\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:15:08.030396 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.030330 2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f60202a-6a6c-45e2-82e3-ade8129035c8-cabundle-cert\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:15:08.030806 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:08.030788 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-whq2k"] Jul 07 15:15:09.310810 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:15:09.310774 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" path="/var/lib/kubelet/pods/5f60202a-6a6c-45e2-82e3-ade8129035c8/volumes" Jul 07 15:16:33.253545 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:16:33.253507 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:16:33.255665 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:16:33.255640 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:19:57.780822 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.780784 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm"] Jul 07 15:19:57.781281 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.781110 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kube-rbac-proxy" Jul 07 15:19:57.781281 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.781124 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kube-rbac-proxy" Jul 07 15:19:57.781281 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.781138 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kserve-container" Jul 07 15:19:57.781281 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.781144 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kserve-container" Jul 07 15:19:57.781281 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.781152 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="storage-initializer" Jul 07 15:19:57.781281 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.781158 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="storage-initializer" Jul 07 15:19:57.781281 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.781208 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kube-rbac-proxy" Jul 07 15:19:57.781281 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.781220 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f60202a-6a6c-45e2-82e3-ade8129035c8" containerName="kserve-container" Jul 07 15:19:57.783605 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.783585 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:57.788865 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.788833 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\"" Jul 07 15:19:57.789007 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.788833 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 15:19:57.789274 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.789255 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 07 15:19:57.789381 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.789280 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 15:19:57.789381 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.789317 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 07 15:19:57.789381 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.789356 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\"" Jul 07 15:19:57.789559 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.789547 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-bnv6j\"" Jul 07 15:19:57.806452 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.806423 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm"] Jul 07 15:19:57.903707 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.903672 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a745134e-0660-4dcb-87ae-dfc67c22e996-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:57.903894 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.903717 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a745134e-0660-4dcb-87ae-dfc67c22e996-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:57.903894 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.903739 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:57.903894 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.903781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fvp\" (UniqueName: \"kubernetes.io/projected/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-api-access-t7fvp\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:57.903894 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:57.903843 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.004774 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.004738 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.004970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.004794 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a745134e-0660-4dcb-87ae-dfc67c22e996-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.004970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.004836 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a745134e-0660-4dcb-87ae-dfc67c22e996-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.004970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.004857 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.004970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.004881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fvp\" (UniqueName: \"kubernetes.io/projected/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-api-access-t7fvp\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.005223 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:19:58.004969 2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found Jul 07 15:19:58.005223 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:19:58.005080 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a745134e-0660-4dcb-87ae-dfc67c22e996-proxy-tls podName:a745134e-0660-4dcb-87ae-dfc67c22e996 nodeName:}" failed. No retries permitted until 2026-07-07 15:19:58.505033791 +0000 UTC m=+1405.713091801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a745134e-0660-4dcb-87ae-dfc67c22e996-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" (UID: "a745134e-0660-4dcb-87ae-dfc67c22e996") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found Jul 07 15:19:58.005340 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.005254 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a745134e-0660-4dcb-87ae-dfc67c22e996-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.005623 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.005601 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.005659 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.005601 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.015729 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.015699 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fvp\" (UniqueName: \"kubernetes.io/projected/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-api-access-t7fvp\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.508915 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.508867 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a745134e-0660-4dcb-87ae-dfc67c22e996-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.511244 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.511226 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a745134e-0660-4dcb-87ae-dfc67c22e996-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.694613 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.694573 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:19:58.832862 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:58.831431 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm"] Jul 07 15:19:59.160822 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:59.160723 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" event={"ID":"a745134e-0660-4dcb-87ae-dfc67c22e996","Type":"ContainerStarted","Data":"8bfa7971f99571bb20c427b44d2b839903c9713177a23bee2d88a82d2f4db3b5"} Jul 07 15:19:59.160822 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:19:59.160802 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" event={"ID":"a745134e-0660-4dcb-87ae-dfc67c22e996","Type":"ContainerStarted","Data":"45677fcada2ef0dc4c36d7793702b6676f48f43912af845353571f66617423c3"} Jul 07 15:20:29.280506 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:20:29.280472 2566 generic.go:356] "Generic (PLEG): container finished" podID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerID="8bfa7971f99571bb20c427b44d2b839903c9713177a23bee2d88a82d2f4db3b5" exitCode=0 Jul 07 15:20:29.280942 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:20:29.280550 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" event={"ID":"a745134e-0660-4dcb-87ae-dfc67c22e996","Type":"ContainerDied","Data":"8bfa7971f99571bb20c427b44d2b839903c9713177a23bee2d88a82d2f4db3b5"} Jul 07 15:20:30.287029 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:20:30.286988 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" event={"ID":"a745134e-0660-4dcb-87ae-dfc67c22e996","Type":"ContainerStarted","Data":"a792b2136e17607e1343b50213f16e45b53340881e39b6ca5e2f9e6485bfe729"} Jul 07 15:20:30.287421 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:20:30.287097 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" event={"ID":"a745134e-0660-4dcb-87ae-dfc67c22e996","Type":"ContainerStarted","Data":"7cc0a690cb2441e150597cfd4603c75fad8862601de4809d4cb7858cf513ffa1"} Jul 07 15:20:30.287421 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:20:30.287396 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:20:30.316219 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:20:30.316152 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" podStartSLOduration=3.64562412 podStartE2EDuration="33.316129852s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.670505732s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:20:30.312563829 +0000 UTC m=+1437.520621875" watchObservedRunningTime="2026-07-07 15:20:30.316129852 +0000 UTC m=+1437.524187885" Jul 07 15:20:31.291858 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:20:31.291809 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:20:37.303277 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:20:37.303235 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:21:33.279237 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:21:33.279210 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:21:33.282360 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:21:33.282334 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:22:02.310737 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:02.310701 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:22:07.849605 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:07.849569 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm"] Jul 07 15:22:07.850161 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:07.849793 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="kserve-container" containerID="cri-o://7cc0a690cb2441e150597cfd4603c75fad8862601de4809d4cb7858cf513ffa1" gracePeriod=30 Jul 07 15:22:07.850161 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:07.849842 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="kube-rbac-proxy" containerID="cri-o://a792b2136e17607e1343b50213f16e45b53340881e39b6ca5e2f9e6485bfe729" gracePeriod=30 Jul 07 15:22:08.685625 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.685587 2566 generic.go:356] "Generic (PLEG): container finished" podID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerID="a792b2136e17607e1343b50213f16e45b53340881e39b6ca5e2f9e6485bfe729" exitCode=2 Jul 07 15:22:08.685625 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.685622 2566 generic.go:356] "Generic (PLEG): container finished" podID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerID="7cc0a690cb2441e150597cfd4603c75fad8862601de4809d4cb7858cf513ffa1" exitCode=0 Jul 07 15:22:08.685789 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.685653 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" event={"ID":"a745134e-0660-4dcb-87ae-dfc67c22e996","Type":"ContainerDied","Data":"a792b2136e17607e1343b50213f16e45b53340881e39b6ca5e2f9e6485bfe729"} Jul 07 15:22:08.685789 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.685738 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" event={"ID":"a745134e-0660-4dcb-87ae-dfc67c22e996","Type":"ContainerDied","Data":"7cc0a690cb2441e150597cfd4603c75fad8862601de4809d4cb7858cf513ffa1"} Jul 07 15:22:08.701832 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.701811 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:22:08.803799 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.803713 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-rbac-proxy-sar-config\") pod \"a745134e-0660-4dcb-87ae-dfc67c22e996\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " Jul 07 15:22:08.803799 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.803782 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7fvp\" (UniqueName: \"kubernetes.io/projected/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-api-access-t7fvp\") pod \"a745134e-0660-4dcb-87ae-dfc67c22e996\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " Jul 07 15:22:08.804032 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.803810 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-cabundle-cert\") pod \"a745134e-0660-4dcb-87ae-dfc67c22e996\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " Jul 07 15:22:08.804032 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.803835 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a745134e-0660-4dcb-87ae-dfc67c22e996-proxy-tls\") pod \"a745134e-0660-4dcb-87ae-dfc67c22e996\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " Jul 07 15:22:08.804032 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.803873 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a745134e-0660-4dcb-87ae-dfc67c22e996-kserve-provision-location\") pod \"a745134e-0660-4dcb-87ae-dfc67c22e996\" (UID: \"a745134e-0660-4dcb-87ae-dfc67c22e996\") " Jul 07 15:22:08.804223 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.804147 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a745134e-0660-4dcb-87ae-dfc67c22e996" (UID: "a745134e-0660-4dcb-87ae-dfc67c22e996"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:22:08.804280 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.804231 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a745134e-0660-4dcb-87ae-dfc67c22e996" (UID: "a745134e-0660-4dcb-87ae-dfc67c22e996"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:22:08.805955 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.805928 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a745134e-0660-4dcb-87ae-dfc67c22e996-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a745134e-0660-4dcb-87ae-dfc67c22e996" (UID: "a745134e-0660-4dcb-87ae-dfc67c22e996"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:22:08.806038 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.805937 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-api-access-t7fvp" (OuterVolumeSpecName: "kube-api-access-t7fvp") pod "a745134e-0660-4dcb-87ae-dfc67c22e996" (UID: "a745134e-0660-4dcb-87ae-dfc67c22e996"). InnerVolumeSpecName "kube-api-access-t7fvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:22:08.809956 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.809928 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a745134e-0660-4dcb-87ae-dfc67c22e996-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a745134e-0660-4dcb-87ae-dfc67c22e996" (UID: "a745134e-0660-4dcb-87ae-dfc67c22e996"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:22:08.904701 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.904666 2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a745134e-0660-4dcb-87ae-dfc67c22e996-kserve-provision-location\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:22:08.904701 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.904698 2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:22:08.904701 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.904711 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7fvp\" (UniqueName: \"kubernetes.io/projected/a745134e-0660-4dcb-87ae-dfc67c22e996-kube-api-access-t7fvp\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:22:08.905171 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.904725 2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a745134e-0660-4dcb-87ae-dfc67c22e996-cabundle-cert\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:22:08.905171 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:08.904739 2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a745134e-0660-4dcb-87ae-dfc67c22e996-proxy-tls\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:22:09.691752 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:09.691655 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" event={"ID":"a745134e-0660-4dcb-87ae-dfc67c22e996","Type":"ContainerDied","Data":"45677fcada2ef0dc4c36d7793702b6676f48f43912af845353571f66617423c3"} Jul 07 15:22:09.691752 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:09.691693 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm" Jul 07 15:22:09.691752 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:09.691742 2566 scope.go:117] "RemoveContainer" containerID="a792b2136e17607e1343b50213f16e45b53340881e39b6ca5e2f9e6485bfe729" Jul 07 15:22:09.699865 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:09.699837 2566 scope.go:117] "RemoveContainer" containerID="7cc0a690cb2441e150597cfd4603c75fad8862601de4809d4cb7858cf513ffa1" Jul 07 15:22:09.707170 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:09.707153 2566 scope.go:117] "RemoveContainer" containerID="8bfa7971f99571bb20c427b44d2b839903c9713177a23bee2d88a82d2f4db3b5" Jul 07 15:22:09.713656 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:09.713634 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm"] Jul 07 15:22:09.718415 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:09.718394 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-lh9bm"] Jul 07 15:22:11.311315 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:11.311282 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" path="/var/lib/kubelet/pods/a745134e-0660-4dcb-87ae-dfc67c22e996/volumes" Jul 07 15:22:13.014570 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.014536 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m"] Jul 07 15:22:13.014970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.014819 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="storage-initializer" Jul 07 15:22:13.014970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.014829 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="storage-initializer" Jul 07 15:22:13.014970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.014841 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="kube-rbac-proxy" Jul 07 15:22:13.014970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.014846 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="kube-rbac-proxy" Jul 07 15:22:13.014970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.014854 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="kserve-container" Jul 07 15:22:13.014970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.014860 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="kserve-container" Jul 07 15:22:13.014970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.014943 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="kserve-container" Jul 07 15:22:13.014970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.014951 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a745134e-0660-4dcb-87ae-dfc67c22e996" containerName="kube-rbac-proxy" Jul 07 15:22:13.018581 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.018565 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.022959 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.022932 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 15:22:13.023108 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.022994 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-bnv6j\"" Jul 07 15:22:13.023108 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.023028 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 15:22:13.024266 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.024242 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 07 15:22:13.024440 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.024408 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\"" Jul 07 15:22:13.024512 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.024440 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 07 15:22:13.024619 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.024601 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\"" Jul 07 15:22:13.029582 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.029560 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m"] Jul 07 15:22:13.141281 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.141244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.141485 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.141298 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjqp\" (UniqueName: \"kubernetes.io/projected/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-api-access-bjjqp\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.141485 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.141378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.141485 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.141462 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.141598 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.141498 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c73ec1-9dab-4e5a-b93c-c30715334ba9-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.242721 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.242675 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c73ec1-9dab-4e5a-b93c-c30715334ba9-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.242884 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.242738 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.242884 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.242773 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjqp\" (UniqueName: \"kubernetes.io/projected/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-api-access-bjjqp\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.242884 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.242804 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.242884 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.242833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.243323 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.243303 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.243627 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.243601 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.243627 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.243619 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.245121 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.245103 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c73ec1-9dab-4e5a-b93c-c30715334ba9-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.252856 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.252831 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjqp\" (UniqueName: \"kubernetes.io/projected/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-api-access-bjjqp\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.329512 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.329429 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:13.472208 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.472181 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m"] Jul 07 15:22:13.474650 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:22:13.474623 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c73ec1_9dab_4e5a_b93c_c30715334ba9.slice/crio-01ff21f4408b64c0c8619d5b42e27bc6302c158061466f5032799458fdc8002a WatchSource:0}: Error finding container 01ff21f4408b64c0c8619d5b42e27bc6302c158061466f5032799458fdc8002a: Status 404 returned error can't find the container with id 01ff21f4408b64c0c8619d5b42e27bc6302c158061466f5032799458fdc8002a Jul 07 15:22:13.716287 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.716194 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" event={"ID":"02c73ec1-9dab-4e5a-b93c-c30715334ba9","Type":"ContainerStarted","Data":"efc09742adea0b68d262020bb74eb5454751389361609ad53805a25f23b1edcf"} Jul 07 15:22:13.716287 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:13.716262 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" event={"ID":"02c73ec1-9dab-4e5a-b93c-c30715334ba9","Type":"ContainerStarted","Data":"01ff21f4408b64c0c8619d5b42e27bc6302c158061466f5032799458fdc8002a"} Jul 07 15:22:33.801179 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:33.801143 2566 generic.go:356] "Generic (PLEG): container finished" podID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerID="efc09742adea0b68d262020bb74eb5454751389361609ad53805a25f23b1edcf" exitCode=0 Jul 07 15:22:33.801179 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:33.801185 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" event={"ID":"02c73ec1-9dab-4e5a-b93c-c30715334ba9","Type":"ContainerDied","Data":"efc09742adea0b68d262020bb74eb5454751389361609ad53805a25f23b1edcf"} Jul 07 15:22:34.809514 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:34.809463 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" event={"ID":"02c73ec1-9dab-4e5a-b93c-c30715334ba9","Type":"ContainerStarted","Data":"f918ee678001ae49a585554264aa59de5809b3b6c44f2115ce3f5a6161b889c6"} Jul 07 15:22:34.810031 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:34.809542 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" event={"ID":"02c73ec1-9dab-4e5a-b93c-c30715334ba9","Type":"ContainerStarted","Data":"20fd0f09c6a532bd3d8359050b8548c2d4696325cd0c8c56873e9d04bb670159"} Jul 07 15:22:34.810031 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:34.809773 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:34.841067 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:34.840997 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" podStartSLOduration=3.368006769 podStartE2EDuration="22.840981666s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.472974897s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:22:34.83950301 +0000 UTC m=+1562.047561043" watchObservedRunningTime="2026-07-07 15:22:34.840981666 +0000 UTC m=+1562.049039699" Jul 07 15:22:35.813575 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:35.813544 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:22:41.823193 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:22:41.823161 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:24:05.315645 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:05.315611 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:24:13.186552 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:13.186510 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m"] Jul 07 15:24:13.186992 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:13.186773 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="kserve-container" containerID="cri-o://20fd0f09c6a532bd3d8359050b8548c2d4696325cd0c8c56873e9d04bb670159" gracePeriod=30 Jul 07 15:24:13.186992 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:13.186826 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="kube-rbac-proxy" containerID="cri-o://f918ee678001ae49a585554264aa59de5809b3b6c44f2115ce3f5a6161b889c6" gracePeriod=30 Jul 07 15:24:14.235444 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.235411 2566 generic.go:356] "Generic (PLEG): container finished" podID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerID="f918ee678001ae49a585554264aa59de5809b3b6c44f2115ce3f5a6161b889c6" exitCode=2 Jul 07 15:24:14.235444 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.235442 2566 generic.go:356] "Generic (PLEG): container finished" podID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerID="20fd0f09c6a532bd3d8359050b8548c2d4696325cd0c8c56873e9d04bb670159" exitCode=0 Jul 07 15:24:14.235830 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.235485 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" event={"ID":"02c73ec1-9dab-4e5a-b93c-c30715334ba9","Type":"ContainerDied","Data":"f918ee678001ae49a585554264aa59de5809b3b6c44f2115ce3f5a6161b889c6"} Jul 07 15:24:14.235830 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.235564 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" event={"ID":"02c73ec1-9dab-4e5a-b93c-c30715334ba9","Type":"ContainerDied","Data":"20fd0f09c6a532bd3d8359050b8548c2d4696325cd0c8c56873e9d04bb670159"} Jul 07 15:24:14.254355 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.254324 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:24:14.355445 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.355336 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kserve-provision-location\") pod \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " Jul 07 15:24:14.355445 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.355392 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjjqp\" (UniqueName: \"kubernetes.io/projected/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-api-access-bjjqp\") pod \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " Jul 07 15:24:14.355445 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.355426 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c73ec1-9dab-4e5a-b93c-c30715334ba9-proxy-tls\") pod \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " Jul 07 15:24:14.355751 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.355457 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-cabundle-cert\") pod \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " Jul 07 15:24:14.355751 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.355477 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-rbac-proxy-sar-config\") pod \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\" (UID: \"02c73ec1-9dab-4e5a-b93c-c30715334ba9\") " Jul 07 15:24:14.356176 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.355887 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02c73ec1-9dab-4e5a-b93c-c30715334ba9" (UID: "02c73ec1-9dab-4e5a-b93c-c30715334ba9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:24:14.356176 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.355962 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02c73ec1-9dab-4e5a-b93c-c30715334ba9" (UID: "02c73ec1-9dab-4e5a-b93c-c30715334ba9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:24:14.357826 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.357793 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c73ec1-9dab-4e5a-b93c-c30715334ba9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02c73ec1-9dab-4e5a-b93c-c30715334ba9" (UID: "02c73ec1-9dab-4e5a-b93c-c30715334ba9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:24:14.357927 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.357832 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-api-access-bjjqp" (OuterVolumeSpecName: "kube-api-access-bjjqp") pod "02c73ec1-9dab-4e5a-b93c-c30715334ba9" (UID: "02c73ec1-9dab-4e5a-b93c-c30715334ba9"). InnerVolumeSpecName "kube-api-access-bjjqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:24:14.360801 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.360763 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02c73ec1-9dab-4e5a-b93c-c30715334ba9" (UID: "02c73ec1-9dab-4e5a-b93c-c30715334ba9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:24:14.457235 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.457185 2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kserve-provision-location\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:14.457235 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.457221 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjjqp\" (UniqueName: \"kubernetes.io/projected/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-api-access-bjjqp\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:14.457235 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.457234 2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c73ec1-9dab-4e5a-b93c-c30715334ba9-proxy-tls\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:14.457235 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.457246 2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-cabundle-cert\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:14.457520 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:14.457255 2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c73ec1-9dab-4e5a-b93c-c30715334ba9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:15.247891 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:15.247857 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" Jul 07 15:24:15.248331 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:15.247851 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m" event={"ID":"02c73ec1-9dab-4e5a-b93c-c30715334ba9","Type":"ContainerDied","Data":"01ff21f4408b64c0c8619d5b42e27bc6302c158061466f5032799458fdc8002a"} Jul 07 15:24:15.248331 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:15.248034 2566 scope.go:117] "RemoveContainer" containerID="f918ee678001ae49a585554264aa59de5809b3b6c44f2115ce3f5a6161b889c6" Jul 07 15:24:15.257627 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:15.257605 2566 scope.go:117] "RemoveContainer" containerID="20fd0f09c6a532bd3d8359050b8548c2d4696325cd0c8c56873e9d04bb670159" Jul 07 15:24:15.266213 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:15.266186 2566 scope.go:117] "RemoveContainer" containerID="efc09742adea0b68d262020bb74eb5454751389361609ad53805a25f23b1edcf" Jul 07 15:24:15.273380 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:15.273345 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m"] Jul 07 15:24:15.275607 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:15.275584 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-cct6m"] Jul 07 15:24:15.310970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:15.310931 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" path="/var/lib/kubelet/pods/02c73ec1-9dab-4e5a-b93c-c30715334ba9/volumes" Jul 07 15:24:18.387833 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.387795 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck"] Jul 07 15:24:18.388302 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.388136 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="kserve-container" Jul 07 15:24:18.388302 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.388148 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="kserve-container" Jul 07 15:24:18.388302 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.388165 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="kube-rbac-proxy" Jul 07 15:24:18.388302 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.388171 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="kube-rbac-proxy" Jul 07 15:24:18.388302 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.388180 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="storage-initializer" Jul 07 15:24:18.388302 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.388186 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="storage-initializer" Jul 07 15:24:18.388302 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.388253 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="kube-rbac-proxy" Jul 07 15:24:18.388302 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.388264 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c73ec1-9dab-4e5a-b93c-c30715334ba9" containerName="kserve-container" Jul 07 15:24:18.393412 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.393386 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.400579 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.400502 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\"" Jul 07 15:24:18.400811 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.400784 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 07 15:24:18.400890 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.400826 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 07 15:24:18.401202 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.401187 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\"" Jul 07 15:24:18.401761 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.401745 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 07 15:24:18.401826 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.401777 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 07 15:24:18.402034 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.402021 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-bnv6j\"" Jul 07 15:24:18.421441 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.421395 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck"] Jul 07 15:24:18.496784 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.496742 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.496784 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.496786 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpsg\" (UniqueName: \"kubernetes.io/projected/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-api-access-tjpsg\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.497114 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.496820 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1451e58e-aeb9-457b-901f-f6be1d1d11e2-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.497114 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.496912 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.497114 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.497028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.598121 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.598028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.598343 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.598178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.598343 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.598218 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.598343 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.598245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpsg\" (UniqueName: \"kubernetes.io/projected/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-api-access-tjpsg\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.598343 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.598284 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1451e58e-aeb9-457b-901f-f6be1d1d11e2-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.598630 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.598602 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.598921 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.598898 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.598970 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.598898 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.600909 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.600884 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1451e58e-aeb9-457b-901f-f6be1d1d11e2-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.610452 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.610418 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpsg\" (UniqueName: \"kubernetes.io/projected/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-api-access-tjpsg\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.704376 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.704262 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:18.856466 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:18.856332 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck"] Jul 07 15:24:18.859188 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:24:18.859154 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1451e58e_aeb9_457b_901f_f6be1d1d11e2.slice/crio-db75a96c627190a3cc393b464197735597e2497b78a0027d2c3c1076488cbce9 WatchSource:0}: Error finding container db75a96c627190a3cc393b464197735597e2497b78a0027d2c3c1076488cbce9: Status 404 returned error can't find the container with id db75a96c627190a3cc393b464197735597e2497b78a0027d2c3c1076488cbce9 Jul 07 15:24:19.268717 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:19.268674 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" event={"ID":"1451e58e-aeb9-457b-901f-f6be1d1d11e2","Type":"ContainerStarted","Data":"7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082"} Jul 07 15:24:19.268951 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:19.268741 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" event={"ID":"1451e58e-aeb9-457b-901f-f6be1d1d11e2","Type":"ContainerStarted","Data":"db75a96c627190a3cc393b464197735597e2497b78a0027d2c3c1076488cbce9"} Jul 07 15:24:29.315505 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:29.315461 2566 generic.go:356] "Generic (PLEG): container finished" podID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerID="7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082" exitCode=0 Jul 07 15:24:29.316028 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:29.315534 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" event={"ID":"1451e58e-aeb9-457b-901f-f6be1d1d11e2","Type":"ContainerDied","Data":"7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082"} Jul 07 15:24:30.322543 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:30.322500 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" event={"ID":"1451e58e-aeb9-457b-901f-f6be1d1d11e2","Type":"ContainerStarted","Data":"9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833"} Jul 07 15:24:30.322943 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:30.322581 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" event={"ID":"1451e58e-aeb9-457b-901f-f6be1d1d11e2","Type":"ContainerStarted","Data":"a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c"} Jul 07 15:24:30.322943 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:30.322781 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:30.351066 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:30.351001 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" podStartSLOduration=2.643823253 podStartE2EDuration="12.350981969s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.707158716s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:24:30.348584384 +0000 UTC m=+1677.556642416" watchObservedRunningTime="2026-07-07 15:24:30.350981969 +0000 UTC m=+1677.559040002" Jul 07 15:24:31.326914 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:31.326874 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:31.328468 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:31.328421 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 07 15:24:32.331776 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:32.331726 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 07 15:24:37.336216 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:37.336184 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:37.336761 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:37.336735 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused" Jul 07 15:24:47.338193 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:47.338154 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:48.440093 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:48.440024 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck"] Jul 07 15:24:48.440573 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:48.440383 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kserve-container" containerID="cri-o://a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c" gracePeriod=30 Jul 07 15:24:48.440573 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:48.440423 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kube-rbac-proxy" containerID="cri-o://9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833" gracePeriod=30 Jul 07 15:24:49.410039 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.410001 2566 generic.go:356] "Generic (PLEG): container finished" podID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerID="9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833" exitCode=2 Jul 07 15:24:49.410235 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.410087 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" event={"ID":"1451e58e-aeb9-457b-901f-f6be1d1d11e2","Type":"ContainerDied","Data":"9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833"} Jul 07 15:24:49.595569 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.595534 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:49.672626 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.672519 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-cabundle-cert\") pod \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " Jul 07 15:24:49.672626 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.672583 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1451e58e-aeb9-457b-901f-f6be1d1d11e2-proxy-tls\") pod \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " Jul 07 15:24:49.672626 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.672624 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-rbac-proxy-sar-config\") pod \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " Jul 07 15:24:49.672950 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.672660 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjpsg\" (UniqueName: \"kubernetes.io/projected/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-api-access-tjpsg\") pod \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " Jul 07 15:24:49.672950 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.672684 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kserve-provision-location\") pod \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\" (UID: \"1451e58e-aeb9-457b-901f-f6be1d1d11e2\") " Jul 07 15:24:49.673103 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.673023 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1451e58e-aeb9-457b-901f-f6be1d1d11e2" (UID: "1451e58e-aeb9-457b-901f-f6be1d1d11e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:24:49.673103 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.673027 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1451e58e-aeb9-457b-901f-f6be1d1d11e2" (UID: "1451e58e-aeb9-457b-901f-f6be1d1d11e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:24:49.673676 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.673652 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1451e58e-aeb9-457b-901f-f6be1d1d11e2" (UID: "1451e58e-aeb9-457b-901f-f6be1d1d11e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:24:49.674912 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.674879 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1451e58e-aeb9-457b-901f-f6be1d1d11e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1451e58e-aeb9-457b-901f-f6be1d1d11e2" (UID: "1451e58e-aeb9-457b-901f-f6be1d1d11e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:24:49.675021 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.674929 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-api-access-tjpsg" (OuterVolumeSpecName: "kube-api-access-tjpsg") pod "1451e58e-aeb9-457b-901f-f6be1d1d11e2" (UID: "1451e58e-aeb9-457b-901f-f6be1d1d11e2"). InnerVolumeSpecName "kube-api-access-tjpsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:24:49.773837 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.773794 2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-cabundle-cert\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:49.773837 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.773829 2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1451e58e-aeb9-457b-901f-f6be1d1d11e2-proxy-tls\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:49.773837 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.773842 2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:49.773837 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.773852 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpsg\" (UniqueName: \"kubernetes.io/projected/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kube-api-access-tjpsg\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:49.774147 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:49.773861 2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1451e58e-aeb9-457b-901f-f6be1d1d11e2-kserve-provision-location\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:24:50.416353 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.416312 2566 generic.go:356] "Generic (PLEG): container finished" podID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerID="a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c" exitCode=0 Jul 07 15:24:50.416557 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.416381 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" Jul 07 15:24:50.416557 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.416402 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" event={"ID":"1451e58e-aeb9-457b-901f-f6be1d1d11e2","Type":"ContainerDied","Data":"a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c"} Jul 07 15:24:50.416557 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.416473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck" event={"ID":"1451e58e-aeb9-457b-901f-f6be1d1d11e2","Type":"ContainerDied","Data":"db75a96c627190a3cc393b464197735597e2497b78a0027d2c3c1076488cbce9"} Jul 07 15:24:50.416557 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.416501 2566 scope.go:117] "RemoveContainer" containerID="9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833" Jul 07 15:24:50.426269 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.426238 2566 scope.go:117] "RemoveContainer" containerID="a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c" Jul 07 15:24:50.435248 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.435214 2566 scope.go:117] "RemoveContainer" containerID="7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082" Jul 07 15:24:50.442905 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.442873 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck"] Jul 07 15:24:50.443963 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.443942 2566 scope.go:117] "RemoveContainer" containerID="9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833" Jul 07 15:24:50.444361 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:24:50.444342 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833\": container with ID starting with 9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833 not found: ID does not exist" containerID="9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833" Jul 07 15:24:50.444439 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.444376 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833"} err="failed to get container status \"9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833\": rpc error: code = NotFound desc = could not find container \"9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833\": container with ID starting with 9fb82a2562b30b442a3526620e7533e4cf15dba1500664ebdf641fc52288d833 not found: ID does not exist" Jul 07 15:24:50.444439 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.444400 2566 scope.go:117] "RemoveContainer" containerID="a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c" Jul 07 15:24:50.444779 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:24:50.444748 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c\": container with ID starting with a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c not found: ID does not exist" containerID="a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c" Jul 07 15:24:50.444906 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.444788 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c"} err="failed to get container status \"a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c\": rpc error: code = NotFound desc = could not find container \"a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c\": container with ID starting with a3cdd6c839fa18cecee2efd38cddd634ce07a410166cb1cb346a3b4b88a8930c not found: ID does not exist" Jul 07 15:24:50.444906 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.444814 2566 scope.go:117] "RemoveContainer" containerID="7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082" Jul 07 15:24:50.445329 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:24:50.445280 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082\": container with ID starting with 7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082 not found: ID does not exist" containerID="7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082" Jul 07 15:24:50.445329 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.445315 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082"} err="failed to get container status \"7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082\": rpc error: code = NotFound desc = could not find container \"7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082\": container with ID starting with 7d019833584cdf2f0c189db0f324276a21f934a6dbdfcd35a291822812481082 not found: ID does not exist" Jul 07 15:24:50.447123 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:50.447096 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-xknck"] Jul 07 15:24:51.314950 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:51.314909 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" path="/var/lib/kubelet/pods/1451e58e-aeb9-457b-901f-f6be1d1d11e2/volumes" Jul 07 15:24:54.616928 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.616874 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6p9hp/must-gather-dhtfb"] Jul 07 15:24:54.617382 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.617258 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="storage-initializer" Jul 07 15:24:54.617382 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.617273 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="storage-initializer" Jul 07 15:24:54.617382 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.617289 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kube-rbac-proxy" Jul 07 15:24:54.617382 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.617294 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kube-rbac-proxy" Jul 07 15:24:54.617382 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.617308 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kserve-container" Jul 07 15:24:54.617382 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.617315 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kserve-container" Jul 07 15:24:54.617382 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.617368 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kserve-container" Jul 07 15:24:54.617382 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.617380 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1451e58e-aeb9-457b-901f-f6be1d1d11e2" containerName="kube-rbac-proxy" Jul 07 15:24:54.622432 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.622402 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:24:54.625506 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.625456 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6p9hp\"/\"default-dockercfg-6sdkq\"" Jul 07 15:24:54.625694 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.625521 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6p9hp\"/\"openshift-service-ca.crt\"" Jul 07 15:24:54.626491 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.626465 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6p9hp\"/\"kube-root-ca.crt\"" Jul 07 15:24:54.630437 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.630334 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6p9hp/must-gather-dhtfb"] Jul 07 15:24:54.713076 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.712998 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ljln\" (UniqueName: \"kubernetes.io/projected/68bcb69b-090e-4893-8189-9dd7c79aff54-kube-api-access-9ljln\") pod \"must-gather-dhtfb\" (UID: \"68bcb69b-090e-4893-8189-9dd7c79aff54\") " pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:24:54.713292 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.713181 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/68bcb69b-090e-4893-8189-9dd7c79aff54-must-gather-output\") pod \"must-gather-dhtfb\" (UID: \"68bcb69b-090e-4893-8189-9dd7c79aff54\") " pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:24:54.814212 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.814167 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ljln\" (UniqueName: \"kubernetes.io/projected/68bcb69b-090e-4893-8189-9dd7c79aff54-kube-api-access-9ljln\") pod \"must-gather-dhtfb\" (UID: \"68bcb69b-090e-4893-8189-9dd7c79aff54\") " pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:24:54.814395 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.814253 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/68bcb69b-090e-4893-8189-9dd7c79aff54-must-gather-output\") pod \"must-gather-dhtfb\" (UID: \"68bcb69b-090e-4893-8189-9dd7c79aff54\") " pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:24:54.814598 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.814578 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/68bcb69b-090e-4893-8189-9dd7c79aff54-must-gather-output\") pod \"must-gather-dhtfb\" (UID: \"68bcb69b-090e-4893-8189-9dd7c79aff54\") " pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:24:54.824551 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.824512 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ljln\" (UniqueName: \"kubernetes.io/projected/68bcb69b-090e-4893-8189-9dd7c79aff54-kube-api-access-9ljln\") pod \"must-gather-dhtfb\" (UID: \"68bcb69b-090e-4893-8189-9dd7c79aff54\") " pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:24:54.933603 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:54.933503 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:24:55.086753 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:55.086678 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6p9hp/must-gather-dhtfb"] Jul 07 15:24:55.089326 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:24:55.089289 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68bcb69b_090e_4893_8189_9dd7c79aff54.slice/crio-ea5e804421e8d99181140da0b71477d57109ac61c55cc694b67f2921e68d486a WatchSource:0}: Error finding container ea5e804421e8d99181140da0b71477d57109ac61c55cc694b67f2921e68d486a: Status 404 returned error can't find the container with id ea5e804421e8d99181140da0b71477d57109ac61c55cc694b67f2921e68d486a Jul 07 15:24:55.091299 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:55.091275 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:24:55.442962 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:24:55.442910 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" event={"ID":"68bcb69b-090e-4893-8189-9dd7c79aff54","Type":"ContainerStarted","Data":"ea5e804421e8d99181140da0b71477d57109ac61c55cc694b67f2921e68d486a"} Jul 07 15:25:02.482542 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:02.482496 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" event={"ID":"68bcb69b-090e-4893-8189-9dd7c79aff54","Type":"ContainerStarted","Data":"8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc"} Jul 07 15:25:02.482542 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:02.482552 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" event={"ID":"68bcb69b-090e-4893-8189-9dd7c79aff54","Type":"ContainerStarted","Data":"2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9"} Jul 07 15:25:02.502254 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:02.502155 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" podStartSLOduration=1.8556674499999999 podStartE2EDuration="8.502133978s" totalImagesPullingTime="6.646466528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:25:02.50062203 +0000 UTC m=+1709.708680062" watchObservedRunningTime="2026-07-07 15:25:02.502133978 +0000 UTC m=+1709.710192010" Jul 07 15:25:23.592720 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:23.592678 2566 generic.go:356] "Generic (PLEG): container finished" podID="68bcb69b-090e-4893-8189-9dd7c79aff54" containerID="2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9" exitCode=0 Jul 07 15:25:23.593185 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:23.592750 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" event={"ID":"68bcb69b-090e-4893-8189-9dd7c79aff54","Type":"ContainerDied","Data":"2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9"} Jul 07 15:25:23.593185 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:23.593138 2566 scope.go:117] "RemoveContainer" containerID="2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9" Jul 07 15:25:24.519849 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:24.519814 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6p9hp_must-gather-dhtfb_68bcb69b-090e-4893-8189-9dd7c79aff54/gather/0.log" Jul 07 15:25:25.078861 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.078813 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2kxcl/must-gather-6v8hl"] Jul 07 15:25:25.081541 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.081518 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kxcl/must-gather-6v8hl" Jul 07 15:25:25.084946 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.084916 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2kxcl\"/\"kube-root-ca.crt\"" Jul 07 15:25:25.086149 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.086126 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2kxcl\"/\"openshift-service-ca.crt\"" Jul 07 15:25:25.086280 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.086164 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2kxcl\"/\"default-dockercfg-kwhgw\"" Jul 07 15:25:25.092323 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.092293 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2kxcl/must-gather-6v8hl"] Jul 07 15:25:25.205283 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.205240 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5172eff4-fd5d-4f80-ba5e-e70d2f559fa7-must-gather-output\") pod \"must-gather-6v8hl\" (UID: \"5172eff4-fd5d-4f80-ba5e-e70d2f559fa7\") " pod="openshift-must-gather-2kxcl/must-gather-6v8hl" Jul 07 15:25:25.205283 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.205288 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vkm\" (UniqueName: \"kubernetes.io/projected/5172eff4-fd5d-4f80-ba5e-e70d2f559fa7-kube-api-access-47vkm\") pod \"must-gather-6v8hl\" (UID: \"5172eff4-fd5d-4f80-ba5e-e70d2f559fa7\") " pod="openshift-must-gather-2kxcl/must-gather-6v8hl" Jul 07 15:25:25.306762 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.306713 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5172eff4-fd5d-4f80-ba5e-e70d2f559fa7-must-gather-output\") pod \"must-gather-6v8hl\" (UID: \"5172eff4-fd5d-4f80-ba5e-e70d2f559fa7\") " pod="openshift-must-gather-2kxcl/must-gather-6v8hl" Jul 07 15:25:25.306961 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.306861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vkm\" (UniqueName: \"kubernetes.io/projected/5172eff4-fd5d-4f80-ba5e-e70d2f559fa7-kube-api-access-47vkm\") pod \"must-gather-6v8hl\" (UID: \"5172eff4-fd5d-4f80-ba5e-e70d2f559fa7\") " pod="openshift-must-gather-2kxcl/must-gather-6v8hl" Jul 07 15:25:25.307208 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.307179 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5172eff4-fd5d-4f80-ba5e-e70d2f559fa7-must-gather-output\") pod \"must-gather-6v8hl\" (UID: \"5172eff4-fd5d-4f80-ba5e-e70d2f559fa7\") " pod="openshift-must-gather-2kxcl/must-gather-6v8hl" Jul 07 15:25:25.316928 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.316890 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vkm\" (UniqueName: \"kubernetes.io/projected/5172eff4-fd5d-4f80-ba5e-e70d2f559fa7-kube-api-access-47vkm\") pod \"must-gather-6v8hl\" (UID: \"5172eff4-fd5d-4f80-ba5e-e70d2f559fa7\") " pod="openshift-must-gather-2kxcl/must-gather-6v8hl" Jul 07 15:25:25.392135 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.392005 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kxcl/must-gather-6v8hl" Jul 07 15:25:25.545876 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.545831 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2kxcl/must-gather-6v8hl"] Jul 07 15:25:25.548999 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:25:25.548961 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5172eff4_fd5d_4f80_ba5e_e70d2f559fa7.slice/crio-9cdca5f32de1c27ea88c1115be27cd317f65c275dba546e312775a958f86d49f WatchSource:0}: Error finding container 9cdca5f32de1c27ea88c1115be27cd317f65c275dba546e312775a958f86d49f: Status 404 returned error can't find the container with id 9cdca5f32de1c27ea88c1115be27cd317f65c275dba546e312775a958f86d49f Jul 07 15:25:25.603736 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:25.603698 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kxcl/must-gather-6v8hl" event={"ID":"5172eff4-fd5d-4f80-ba5e-e70d2f559fa7","Type":"ContainerStarted","Data":"9cdca5f32de1c27ea88c1115be27cd317f65c275dba546e312775a958f86d49f"} Jul 07 15:25:26.612073 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:26.611626 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kxcl/must-gather-6v8hl" event={"ID":"5172eff4-fd5d-4f80-ba5e-e70d2f559fa7","Type":"ContainerStarted","Data":"7386aca3e43b4945add6ef500d4b6e9a53a63b8a2ee6fd66d642160a7092b7d6"} Jul 07 15:25:27.621142 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:27.621093 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kxcl/must-gather-6v8hl" event={"ID":"5172eff4-fd5d-4f80-ba5e-e70d2f559fa7","Type":"ContainerStarted","Data":"935764ab5b9280664bfa004196e1321a366ca6da39b3a6594205fed7740d4c6d"} Jul 07 15:25:27.643198 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:27.643119 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2kxcl/must-gather-6v8hl" podStartSLOduration=1.744797535 podStartE2EDuration="2.643089479s" totalImagesPullingTime="898.291944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:25:27.64038695 +0000 UTC m=+1734.848444997" watchObservedRunningTime="2026-07-07 15:25:27.643089479 +0000 UTC m=+1734.851147512" Jul 07 15:25:28.172601 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:28.172500 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kwswc_9c09a0b2-4116-49e0-9727-aa0de72a0b22/global-pull-secret-syncer/0.log" Jul 07 15:25:28.381504 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:28.381469 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xckkk_9c03f5dc-fe5d-4940-8106-92a596ff68e6/konnectivity-agent/0.log" Jul 07 15:25:28.409699 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:28.409653 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-13.ec2.internal_952eb1cd57023c1b0bde0ddde055ed58/haproxy/0.log" Jul 07 15:25:29.932697 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:29.932639 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6p9hp/must-gather-dhtfb"] Jul 07 15:25:29.933253 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:29.932902 2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" containerName="copy" containerID="cri-o://8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc" gracePeriod=2 Jul 07 15:25:29.940413 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:29.940376 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6p9hp/must-gather-dhtfb"] Jul 07 15:25:30.343553 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.343415 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6p9hp_must-gather-dhtfb_68bcb69b-090e-4893-8189-9dd7c79aff54/copy/0.log" Jul 07 15:25:30.348158 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.347780 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:25:30.351149 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:25:30.351088 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dhtfb\" is forbidden: User \"system:node:ip-10-0-130-13.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6p9hp\": no relationship found between node 'ip-10-0-130-13.ec2.internal' and this object" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:25:30.459404 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.456807 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/68bcb69b-090e-4893-8189-9dd7c79aff54-must-gather-output\") pod \"68bcb69b-090e-4893-8189-9dd7c79aff54\" (UID: \"68bcb69b-090e-4893-8189-9dd7c79aff54\") " Jul 07 15:25:30.459404 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.456901 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ljln\" (UniqueName: \"kubernetes.io/projected/68bcb69b-090e-4893-8189-9dd7c79aff54-kube-api-access-9ljln\") pod \"68bcb69b-090e-4893-8189-9dd7c79aff54\" (UID: \"68bcb69b-090e-4893-8189-9dd7c79aff54\") " Jul 07 15:25:30.459404 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.458959 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bcb69b-090e-4893-8189-9dd7c79aff54-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "68bcb69b-090e-4893-8189-9dd7c79aff54" (UID: "68bcb69b-090e-4893-8189-9dd7c79aff54"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:25:30.463143 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.462220 2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bcb69b-090e-4893-8189-9dd7c79aff54-kube-api-access-9ljln" (OuterVolumeSpecName: "kube-api-access-9ljln") pod "68bcb69b-090e-4893-8189-9dd7c79aff54" (UID: "68bcb69b-090e-4893-8189-9dd7c79aff54"). InnerVolumeSpecName "kube-api-access-9ljln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:25:30.558249 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.558096 2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/68bcb69b-090e-4893-8189-9dd7c79aff54-must-gather-output\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:25:30.558249 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.558161 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ljln\" (UniqueName: \"kubernetes.io/projected/68bcb69b-090e-4893-8189-9dd7c79aff54-kube-api-access-9ljln\") on node \"ip-10-0-130-13.ec2.internal\" DevicePath \"\"" Jul 07 15:25:30.641980 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.641920 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6p9hp_must-gather-dhtfb_68bcb69b-090e-4893-8189-9dd7c79aff54/copy/0.log" Jul 07 15:25:30.642672 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.642631 2566 generic.go:356] "Generic (PLEG): container finished" podID="68bcb69b-090e-4893-8189-9dd7c79aff54" containerID="8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc" exitCode=143 Jul 07 15:25:30.642913 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.642891 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:25:30.643102 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.643079 2566 scope.go:117] "RemoveContainer" containerID="8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc" Jul 07 15:25:30.646995 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:25:30.646951 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dhtfb\" is forbidden: User \"system:node:ip-10-0-130-13.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6p9hp\": no relationship found between node 'ip-10-0-130-13.ec2.internal' and this object" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:25:30.659915 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.659876 2566 scope.go:117] "RemoveContainer" containerID="2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9" Jul 07 15:25:30.683072 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:25:30.681455 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dhtfb\" is forbidden: User \"system:node:ip-10-0-130-13.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6p9hp\": no relationship found between node 'ip-10-0-130-13.ec2.internal' and this object" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" pod="openshift-must-gather-6p9hp/must-gather-dhtfb" Jul 07 15:25:30.700292 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.692846 2566 scope.go:117] "RemoveContainer" containerID="8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc" Jul 07 15:25:30.700292 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:25:30.693672 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc\": container with ID starting with 8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc not found: ID does not exist" containerID="8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc" Jul 07 15:25:30.700292 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.693730 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc"} err="failed to get container status \"8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc\": rpc error: code = NotFound desc = could not find container \"8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc\": container with ID starting with 8bc7d0192d2da47659718582ed32bdda1046049515d720a22edc21d914ec37dc not found: ID does not exist" Jul 07 15:25:30.700292 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.693772 2566 scope.go:117] "RemoveContainer" containerID="2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9" Jul 07 15:25:30.700292 ip-10-0-130-13 kubenswrapper[2566]: E0707 15:25:30.695422 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9\": container with ID starting with 2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9 not found: ID does not exist" containerID="2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9" Jul 07 15:25:30.700292 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:30.695491 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9"} err="failed to get container status \"2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9\": rpc error: code = NotFound desc = could not find container \"2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9\": container with ID starting with 2de5e5d43d3db3d70c911cbe840261538b75010799f4628f3e65e84ad0da9eb9 not found: ID does not exist" Jul 07 15:25:31.316239 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:31.316186 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" path="/var/lib/kubelet/pods/68bcb69b-090e-4893-8189-9dd7c79aff54/volumes" Jul 07 15:25:33.914034 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:33.913912 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pl58j_f5f948de-d5b3-4fa1-a29a-b43803289d62/node-exporter/0.log" Jul 07 15:25:33.940907 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:33.940876 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pl58j_f5f948de-d5b3-4fa1-a29a-b43803289d62/kube-rbac-proxy/0.log" Jul 07 15:25:33.969749 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:33.969716 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pl58j_f5f948de-d5b3-4fa1-a29a-b43803289d62/init-textfile/0.log" Jul 07 15:25:34.119640 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.119610 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-68fkx_6554ecd7-3748-40bf-a291-bd0437e6c973/kube-rbac-proxy-main/0.log" Jul 07 15:25:34.151953 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.151837 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-68fkx_6554ecd7-3748-40bf-a291-bd0437e6c973/kube-rbac-proxy-self/0.log" Jul 07 15:25:34.178602 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.178448 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-68fkx_6554ecd7-3748-40bf-a291-bd0437e6c973/openshift-state-metrics/0.log" Jul 07 15:25:34.231010 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.230967 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4ab7024d-dd8d-441c-860c-900a27b8aa53/prometheus/0.log" Jul 07 15:25:34.256628 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.256589 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4ab7024d-dd8d-441c-860c-900a27b8aa53/config-reloader/0.log" Jul 07 15:25:34.290770 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.290734 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4ab7024d-dd8d-441c-860c-900a27b8aa53/thanos-sidecar/0.log" Jul 07 15:25:34.320193 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.320165 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4ab7024d-dd8d-441c-860c-900a27b8aa53/kube-rbac-proxy-web/0.log" Jul 07 15:25:34.348708 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.348663 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4ab7024d-dd8d-441c-860c-900a27b8aa53/kube-rbac-proxy/0.log" Jul 07 15:25:34.378903 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.378868 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4ab7024d-dd8d-441c-860c-900a27b8aa53/kube-rbac-proxy-thanos/0.log" Jul 07 15:25:34.409655 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.409621 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4ab7024d-dd8d-441c-860c-900a27b8aa53/init-config-reloader/0.log" Jul 07 15:25:34.451881 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.451760 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-v6x9v_5734779e-8983-4e75-9a6e-4787d908cb85/prometheus-operator/0.log" Jul 07 15:25:34.478343 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.478312 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-v6x9v_5734779e-8983-4e75-9a6e-4787d908cb85/kube-rbac-proxy/0.log" Jul 07 15:25:34.510573 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.510536 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-6pfxb_43c78cfa-7ffd-444f-ad38-29847a38ba9a/prometheus-operator-admission-webhook/0.log" Jul 07 15:25:34.575567 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.575534 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fb4b857b6-rjfqm_5b0737e2-9920-4473-ac78-8844f7e46d21/telemeter-client/0.log" Jul 07 15:25:34.615180 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.615152 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fb4b857b6-rjfqm_5b0737e2-9920-4473-ac78-8844f7e46d21/reload/0.log" Jul 07 15:25:34.656951 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:34.656915 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fb4b857b6-rjfqm_5b0737e2-9920-4473-ac78-8844f7e46d21/kube-rbac-proxy/0.log" Jul 07 15:25:36.867410 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:36.867369 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/1.log" Jul 07 15:25:36.873793 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:36.873757 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8lrt5_24927288-0653-428f-98cf-c7c53e70ab43/console-operator/2.log" Jul 07 15:25:37.026860 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.026823 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw"] Jul 07 15:25:37.027255 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.027224 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" containerName="gather" Jul 07 15:25:37.027255 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.027253 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" containerName="gather" Jul 07 15:25:37.027419 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.027273 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" containerName="copy" Jul 07 15:25:37.027419 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.027282 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" containerName="copy" Jul 07 15:25:37.027419 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.027343 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" containerName="gather" Jul 07 15:25:37.027419 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.027358 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bcb69b-090e-4893-8189-9dd7c79aff54" containerName="copy" Jul 07 15:25:37.031894 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.031858 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.040360 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.040321 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw"] Jul 07 15:25:37.224127 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.224019 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-podres\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.224127 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.224109 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45z2g\" (UniqueName: \"kubernetes.io/projected/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-kube-api-access-45z2g\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.224522 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.224152 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-proc\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.224522 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.224202 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-lib-modules\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.224522 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.224219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-sys\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.325650 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.325612 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-sys\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.325835 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.325711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-podres\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.325835 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.325742 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-sys\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.325835 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.325762 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45z2g\" (UniqueName: \"kubernetes.io/projected/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-kube-api-access-45z2g\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.325958 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.325852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-proc\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.325958 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.325897 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-lib-modules\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.325958 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.325930 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-podres\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.326097 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.325961 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-proc\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.326097 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.326031 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-lib-modules\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.335316 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.335273 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45z2g\" (UniqueName: \"kubernetes.io/projected/bf74f984-4a97-4ebb-b9af-ef0b8f41b813-kube-api-access-45z2g\") pod \"perf-node-gather-daemonset-msztw\" (UID: \"bf74f984-4a97-4ebb-b9af-ef0b8f41b813\") " pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.346988 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.346940 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.384079 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.384018 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-9nghg_68f1313e-1e49-47e6-89af-a00a13e35f38/download-server/0.log" Jul 07 15:25:37.545078 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.543454 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw"] Jul 07 15:25:37.547826 ip-10-0-130-13 kubenswrapper[2566]: W0707 15:25:37.547779 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbf74f984_4a97_4ebb_b9af_ef0b8f41b813.slice/crio-9e112099b21fea67e28cb742300b42e03c3b132c158a83db8a2ae9829c1f677e WatchSource:0}: Error finding container 9e112099b21fea67e28cb742300b42e03c3b132c158a83db8a2ae9829c1f677e: Status 404 returned error can't find the container with id 9e112099b21fea67e28cb742300b42e03c3b132c158a83db8a2ae9829c1f677e Jul 07 15:25:37.689936 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.689896 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" event={"ID":"bf74f984-4a97-4ebb-b9af-ef0b8f41b813","Type":"ContainerStarted","Data":"6ab7a9d785c856bdec6d9fbe5f9a30a7507ffcc58b307cfb66b27017122feb4d"} Jul 07 15:25:37.690173 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.689964 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" event={"ID":"bf74f984-4a97-4ebb-b9af-ef0b8f41b813","Type":"ContainerStarted","Data":"9e112099b21fea67e28cb742300b42e03c3b132c158a83db8a2ae9829c1f677e"} Jul 07 15:25:37.690173 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.690017 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:37.711510 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:37.711441 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" podStartSLOduration=0.711420264 podStartE2EDuration="711.420264ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:25:37.710208779 +0000 UTC m=+1744.918266813" watchObservedRunningTime="2026-07-07 15:25:37.711420264 +0000 UTC m=+1744.919478295" Jul 07 15:25:39.119678 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:39.119651 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lq95g_a7397841-807c-42d8-b55f-920d4adeb369/dns/0.log" Jul 07 15:25:39.147717 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:39.147686 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lq95g_a7397841-807c-42d8-b55f-920d4adeb369/kube-rbac-proxy/0.log" Jul 07 15:25:39.306667 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:39.306633 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f74m_ddcd8fcc-a9ff-4a58-85b1-b6b5770eceac/dns-node-resolver/0.log" Jul 07 15:25:39.965518 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:39.965482 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dlvgv_627d2b1b-5d4b-416b-ad66-687277f43add/node-ca/0.log" Jul 07 15:25:41.332148 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:41.332114 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nmchs_c36255e9-35a1-4c63-8154-aaa712e37e96/serve-healthcheck-canary/0.log" Jul 07 15:25:41.780022 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:41.779991 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-kqd56_171153e3-a823-4d76-a913-f83499139e27/insights-operator/0.log" Jul 07 15:25:41.780944 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:41.780899 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-kqd56_171153e3-a823-4d76-a913-f83499139e27/insights-operator/1.log" Jul 07 15:25:41.876621 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:41.876591 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-csrnj_4479f7df-b08d-45b4-9c23-dfef45423cb1/kube-rbac-proxy/0.log" Jul 07 15:25:41.900353 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:41.900318 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-csrnj_4479f7df-b08d-45b4-9c23-dfef45423cb1/exporter/0.log" Jul 07 15:25:41.924717 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:41.924661 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-csrnj_4479f7df-b08d-45b4-9c23-dfef45423cb1/extractor/0.log" Jul 07 15:25:43.711959 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:43.711923 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2kxcl/perf-node-gather-daemonset-msztw" Jul 07 15:25:44.357866 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:44.357834 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-wbkwl_211ee1ce-2f3d-4d72-820f-58ff699a3624/server/0.log" Jul 07 15:25:44.503173 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:44.503138 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9twvw_47fe24a4-1beb-4cbd-8423-475fba20ebe2/s3-init/0.log" Jul 07 15:25:44.522964 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:44.522931 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9twvw_47fe24a4-1beb-4cbd-8423-475fba20ebe2/download-hf-model/0.log" Jul 07 15:25:44.554420 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:44.554390 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7gktl_f2132551-2e35-43c7-9c0e-793ed23b2920/s3-tls-init-custom/0.log" Jul 07 15:25:44.574865 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:44.574834 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7gktl_f2132551-2e35-43c7-9c0e-793ed23b2920/download-hf-model/0.log" Jul 07 15:25:44.607600 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:44.607563 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-9zsf7_2db8ebce-37ce-4e5e-b012-f2da9772befb/s3-tls-init-serving/0.log" Jul 07 15:25:44.626696 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:44.626581 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-9zsf7_2db8ebce-37ce-4e5e-b012-f2da9772befb/download-hf-model/0.log" Jul 07 15:25:51.228064 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.228018 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mn7mm_92432074-635b-4c94-a0b9-363d83ed7127/kube-multus-additional-cni-plugins/0.log" Jul 07 15:25:51.259165 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.259136 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mn7mm_92432074-635b-4c94-a0b9-363d83ed7127/egress-router-binary-copy/0.log" Jul 07 15:25:51.283330 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.283296 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mn7mm_92432074-635b-4c94-a0b9-363d83ed7127/cni-plugins/0.log" Jul 07 15:25:51.306940 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.306903 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mn7mm_92432074-635b-4c94-a0b9-363d83ed7127/bond-cni-plugin/0.log" Jul 07 15:25:51.329826 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.329789 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mn7mm_92432074-635b-4c94-a0b9-363d83ed7127/routeoverride-cni/0.log" Jul 07 15:25:51.356088 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.356058 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mn7mm_92432074-635b-4c94-a0b9-363d83ed7127/whereabouts-cni-bincopy/0.log" Jul 07 15:25:51.378580 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.378550 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mn7mm_92432074-635b-4c94-a0b9-363d83ed7127/whereabouts-cni/0.log" Jul 07 15:25:51.582945 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.582914 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cfcxp_ea558fad-42cc-435d-aeff-cdb5ada41261/kube-multus/0.log" Jul 07 15:25:51.656361 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.656323 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jw459_f115fbf4-80c0-4991-b663-61ce83d6e968/network-metrics-daemon/0.log" Jul 07 15:25:51.678359 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:51.678329 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jw459_f115fbf4-80c0-4991-b663-61ce83d6e968/kube-rbac-proxy/0.log" Jul 07 15:25:52.520360 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:52.520305 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw2hn_0678b753-d3d1-4fd9-950d-eec1f33fcc86/ovn-controller/0.log" Jul 07 15:25:52.541099 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:52.541062 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw2hn_0678b753-d3d1-4fd9-950d-eec1f33fcc86/ovn-acl-logging/0.log" Jul 07 15:25:52.572173 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:52.572145 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw2hn_0678b753-d3d1-4fd9-950d-eec1f33fcc86/kube-rbac-proxy-node/0.log" Jul 07 15:25:52.595791 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:52.595756 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw2hn_0678b753-d3d1-4fd9-950d-eec1f33fcc86/kube-rbac-proxy-ovn-metrics/0.log" Jul 07 15:25:52.619026 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:52.618991 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw2hn_0678b753-d3d1-4fd9-950d-eec1f33fcc86/northd/0.log" Jul 07 15:25:52.645553 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:52.645522 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw2hn_0678b753-d3d1-4fd9-950d-eec1f33fcc86/nbdb/0.log" Jul 07 15:25:52.675607 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:52.675575 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw2hn_0678b753-d3d1-4fd9-950d-eec1f33fcc86/sbdb/0.log" Jul 07 15:25:52.725531 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:52.725469 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw2hn_0678b753-d3d1-4fd9-950d-eec1f33fcc86/ovnkube-controller/0.log" Jul 07 15:25:54.391238 ip-10-0-130-13 kubenswrapper[2566]: I0707 15:25:54.391194 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mvn5b_0c6588bc-27c9-491e-aac7-b6b1597fdfe0/network-check-target-container/0.log"