Jul 10 20:06:24.603031 ip-10-0-143-108 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 20:06:24.603047 ip-10-0-143-108 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 20:06:24.603059 ip-10-0-143-108 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 20:06:24.603666 ip-10-0-143-108 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 20:06:34.644338 ip-10-0-143-108 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 20:06:34.644362 ip-10-0-143-108 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 14a4004645414b9f8d76ff26c9370538 -- Jul 10 20:08:39.415202 ip-10-0-143-108 systemd[1]: Starting Kubernetes Kubelet... Jul 10 20:08:39.785302 ip-10-0-143-108 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 20:08:39.785302 ip-10-0-143-108 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 20:08:39.785302 ip-10-0-143-108 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 20:08:39.785302 ip-10-0-143-108 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 20:08:39.785302 ip-10-0-143-108 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 20:08:39.786529 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.786122 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 20:08:39.791544 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791514 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:08:39.791544 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791539 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:08:39.791544 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791544 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:08:39.791544 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791547 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:08:39.791544 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791550 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:08:39.791544 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791553 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791556 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791560 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791562 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791565 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791568 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791571 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791573 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791576 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791579 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791582 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791584 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791586 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791589 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791592 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791594 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791597 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791599 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791601 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791604 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:08:39.791833 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791607 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791609 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791612 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791618 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791620 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791623 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791625 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791627 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791630 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791632 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791636 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791638 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791640 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791645 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791649 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791651 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791654 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791657 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791659 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:08:39.792334 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791662 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791664 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791667 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791669 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791671 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791674 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791677 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791680 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791682 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791685 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791687 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791690 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791693 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791695 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791698 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791701 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791703 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791705 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791708 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791710 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:08:39.792901 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791712 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791715 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791717 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791720 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791723 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791726 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791729 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791731 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791733 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791736 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791739 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791741 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791744 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791746 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791750 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791754 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791758 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791761 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791763 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:08:39.793387 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791768 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791771 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791774 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791776 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791780 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791782 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791785 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791789 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791792 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791794 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791797 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791800 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791802 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791805 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791807 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.791809 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792564 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792571 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792575 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792578 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:08:39.793836 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792581 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792583 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792586 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792588 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792591 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792594 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792596 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792599 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792601 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792604 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792607 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792609 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792612 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792615 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792618 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792621 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792624 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792626 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792629 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792631 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:08:39.794338 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792634 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792636 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792639 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792641 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792644 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792646 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792649 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792651 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792655 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792659 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792662 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792665 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792668 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792671 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792674 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792676 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792679 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792681 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792684 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:08:39.794878 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792687 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792689 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792692 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792694 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792697 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792699 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792702 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792705 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792710 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792712 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792715 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792717 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792720 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792723 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792726 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792729 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792731 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792734 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792737 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792740 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:08:39.795348 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792742 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792745 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792749 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792751 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792754 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792756 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792759 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792762 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792764 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792767 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792769 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792772 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792774 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792777 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792779 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792782 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792784 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792787 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792789 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792792 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:08:39.795829 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792796 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792799 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792801 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792805 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792808 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792810 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792813 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792816 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792820 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792823 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792826 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792829 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792831 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792834 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792836 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.792839 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793312 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793318 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793321 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:08:39.796347 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793324 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793327 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793329 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793332 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793335 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793338 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793340 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793342 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793345 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793347 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793350 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793352 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793355 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793359 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793361 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793364 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793366 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793369 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793372 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793375 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:08:39.796814 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793378 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793380 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793383 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793385 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793388 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793391 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793394 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793396 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793399 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793402 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793404 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793407 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793409 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793412 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793414 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793418 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793422 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793424 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793427 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:08:39.797311 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793430 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793432 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793435 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793437 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793440 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793442 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793445 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793448 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793452 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793455 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793457 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793460 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793463 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793465 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793468 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793470 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793472 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793475 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793477 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:08:39.797771 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793480 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793483 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793487 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793490 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793492 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793495 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793497 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793499 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793502 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793504 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793506 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793509 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793511 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793514 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793516 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793518 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793520 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793523 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793525 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793529 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:08:39.798251 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793531 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793534 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793537 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793540 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793543 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793547 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793551 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793554 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793557 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793559 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793562 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793564 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793566 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793569 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793571 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793574 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793577 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.793579 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793664 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793672 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 20:08:39.798736 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793684 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793689 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793693 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793696 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793700 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793704 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793707 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793710 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793714 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793717 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793720 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793723 2565 flags.go:64] FLAG: --cgroup-root="" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793726 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793730 2565 flags.go:64] FLAG: --client-ca-file="" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793734 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793740 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793745 2565 flags.go:64] FLAG: --cluster-domain="" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793747 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793751 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793753 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793757 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793760 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793763 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 20:08:39.799232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793766 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793770 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793773 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793776 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793779 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793782 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793785 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793788 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793792 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793795 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793797 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793800 2565 flags.go:64] FLAG: --enable-server="true" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793803 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793807 2565 flags.go:64] FLAG: --event-burst="100" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793810 2565 flags.go:64] FLAG: --event-qps="50" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793813 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793816 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793818 2565 flags.go:64] FLAG: --eviction-hard="" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793822 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793827 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793830 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793833 2565 flags.go:64] FLAG: --eviction-soft="" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793836 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793839 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793848 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 20:08:39.799808 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793852 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793855 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793872 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793875 2565 flags.go:64] FLAG: --feature-gates="" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793880 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793883 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793886 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793889 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793891 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793895 2565 flags.go:64] FLAG: --help="false" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793898 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-143-108.ec2.internal" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793901 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793904 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793907 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793910 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793913 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793916 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793919 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793922 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793924 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793927 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793930 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793933 2565 flags.go:64] FLAG: --kube-reserved="" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793936 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 20:08:39.800503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793939 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793942 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793946 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793949 2565 flags.go:64] FLAG: --lock-file="" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793951 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793954 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793957 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793963 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793967 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793970 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793972 2565 flags.go:64] FLAG: --logging-format="text" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793975 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793978 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793981 2565 flags.go:64] FLAG: --manifest-url="" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793984 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793988 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793991 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793995 2565 flags.go:64] FLAG: --max-pods="110" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.793998 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794001 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794003 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794006 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794009 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794012 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794015 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 20:08:39.801108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794022 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794025 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794028 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794030 2565 flags.go:64] FLAG: --pod-cidr="" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794033 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794038 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794041 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794044 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794047 2565 flags.go:64] FLAG: --port="10250" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794050 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794053 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04c3bfe444f31e2f6" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794056 2565 flags.go:64] FLAG: --qos-reserved="" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794059 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794062 2565 flags.go:64] FLAG: --register-node="true" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794064 2565 flags.go:64] FLAG: --register-with-taints="" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794069 2565 flags.go:64] FLAG: --registry-burst="10" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794073 2565 flags.go:64] FLAG: --registry-qps="5" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794076 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794079 2565 flags.go:64] FLAG: --reserved-memory="" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794082 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794085 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794088 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794091 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794094 2565 flags.go:64] FLAG: --runonce="false" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794097 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 20:08:39.801730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794100 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794103 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794106 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794108 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794111 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794114 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794117 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794120 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794123 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794126 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794129 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794132 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794135 2565 flags.go:64] FLAG: --system-cgroups="" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794138 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794143 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794146 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794148 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794154 2565 flags.go:64] FLAG: --tls-min-version="" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794157 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794159 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794162 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794165 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794168 2565 flags.go:64] FLAG: --v="2" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794172 2565 flags.go:64] FLAG: --version="false" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794177 2565 flags.go:64] FLAG: --vmodule="" Jul 10 20:08:39.802363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794181 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.794184 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794310 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794314 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794317 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794320 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794331 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794334 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794337 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794340 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794342 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794345 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794348 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794350 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794353 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794356 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794359 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794362 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794364 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794367 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:08:39.802985 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794369 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794372 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794374 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794377 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794379 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794381 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794384 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794387 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794389 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794391 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794394 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794396 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794402 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794405 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794408 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794410 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794414 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794417 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794420 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:08:39.803511 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794424 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794427 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794430 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794433 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794435 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794437 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794441 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794444 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794447 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794449 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794452 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794454 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794457 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794459 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794462 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794464 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794467 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794469 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794472 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794474 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:08:39.804010 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794478 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794480 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794483 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794485 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794488 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794490 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794493 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794496 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794499 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794501 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794503 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794506 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794510 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794512 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794515 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794517 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794519 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794522 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794525 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794527 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:08:39.804495 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794530 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794532 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794535 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794537 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794539 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794542 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794544 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794548 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794551 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794555 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794557 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794560 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794563 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794566 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794569 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794571 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794575 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794577 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794580 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794583 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:08:39.804997 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794586 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.794588 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.795179 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.802258 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.802278 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.802957 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.802970 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803043 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803049 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803052 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803055 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803058 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803061 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803063 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803066 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:08:39.805488 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803068 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803071 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803075 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803078 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803081 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803084 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803086 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803089 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803091 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803093 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803096 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803098 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803101 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803104 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803107 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803109 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803112 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803114 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803117 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:08:39.805856 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803119 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803122 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803132 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803135 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803138 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803142 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803146 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803149 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803152 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803155 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803158 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803161 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803165 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803168 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803171 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803174 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803177 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803180 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803183 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:08:39.806344 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803185 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803188 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803191 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803194 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803196 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803199 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803201 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803204 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803206 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803209 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803211 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803213 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803216 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803218 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803221 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803224 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803233 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803236 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803239 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803241 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:08:39.806820 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803243 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803246 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803248 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803251 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803253 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803256 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803258 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803261 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803263 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803265 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803268 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803270 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803273 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803275 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803277 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803280 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803282 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803284 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803287 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803289 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:08:39.807362 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803292 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803294 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803296 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803299 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803301 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803303 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803306 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803309 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803311 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803321 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803324 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803326 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803329 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.803334 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 20:08:39.807835 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803483 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803490 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803493 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803495 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803498 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803501 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803504 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803506 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803509 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803512 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803515 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803517 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803520 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803523 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803526 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803529 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803531 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803534 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803536 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803539 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 20:08:39.808292 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803541 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803544 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803546 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803549 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803551 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803554 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803557 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803559 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803572 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803575 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803578 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803580 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803582 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803585 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803587 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803590 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803592 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803595 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803597 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803600 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 20:08:39.808763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803602 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803604 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803607 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803610 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803612 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803614 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803617 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803621 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803624 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803628 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803632 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803635 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803638 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803641 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803644 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803646 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803649 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803652 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803655 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 20:08:39.809280 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803657 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803660 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803679 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803682 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803684 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803687 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803689 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803692 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803694 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803697 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803699 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803701 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803704 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803706 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803709 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803711 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803714 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803716 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803719 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803722 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 20:08:39.809763 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803724 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803727 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803731 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803735 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803738 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803741 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803744 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803746 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803749 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803752 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803755 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803757 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803760 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803762 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803784 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803794 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803797 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803801 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803803 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 20:08:39.810270 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:39.803806 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 20:08:39.810761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.803811 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 20:08:39.810761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.803983 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 20:08:39.810761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.806083 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 20:08:39.810761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.807504 2565 server.go:1019] "Starting client certificate rotation" Jul 10 20:08:39.810761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.807607 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 20:08:39.810761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.807644 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 20:08:39.810761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.810631 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 20:08:39.815143 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.815085 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 20:08:39.824478 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.824455 2565 log.go:25] "Validated CRI v1 runtime API" Jul 10 20:08:39.826245 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.826229 2565 log.go:25] "Validated CRI v1 image API" Jul 10 20:08:39.827018 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.826999 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 20:08:39.830409 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.830377 2565 fs.go:135] Filesystem UUIDs: map[08f2e6b2-d01e-410f-8274-0853e6c8f787:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8f9d6ffd-2ec3-462f-b7b3-302ec02b8096:/dev/nvme0n1p4] Jul 10 20:08:39.830493 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.830406 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 20:08:39.835711 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.835584 2565 manager.go:216] Machine: {Timestamp:2026-07-10 20:08:39.834218428 +0000 UTC m=+0.307187745 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090045 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea6c8eb3e30580055a3f65f74841d SystemUUID:ec2ea6c8-eb3e-3058-0055-a3f65f74841d BootID:14a40046-4541-4b9f-8d76-ff26c9370538 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:13:c2:07:5d:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:13:c2:07:5d:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:ba:9e:7f:8d:a9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 20:08:39.835711 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.835701 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 20:08:39.835920 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.835825 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 20:08:39.838607 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.838579 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 20:08:39.838770 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.838611 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-108.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 20:08:39.839378 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.839364 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 20:08:39.839420 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.839381 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 20:08:39.839420 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.839389 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 20:08:39.840155 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.840141 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 20:08:39.840936 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.840924 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 20:08:39.843702 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.843687 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 10 20:08:39.843832 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.843822 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 20:08:39.843883 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.843856 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 10 20:08:39.843928 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.843886 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 20:08:39.843928 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.843902 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 20:08:39.843928 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.843911 2565 kubelet.go:398] "Adding apiserver pod source" Jul 10 20:08:39.843928 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.843921 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 20:08:39.844346 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.844333 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 20:08:39.844386 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.844353 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 20:08:39.845298 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.845279 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 20:08:39.847552 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.847534 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 20:08:39.848987 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.848973 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 20:08:39.849030 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.849004 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.849958 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.849989 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850001 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850013 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850031 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850040 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850054 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850069 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850081 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850090 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850103 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850111 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850810 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 20:08:39.851094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.850830 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 20:08:39.854443 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.854423 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-108.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 20:08:39.854526 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.854432 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 20:08:39.854597 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.854561 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 20:08:39.855733 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.855719 2565 server.go:1268] "Started kubelet" Jul 10 20:08:39.855814 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.855786 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 20:08:39.855895 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.855807 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 20:08:39.855935 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.855896 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 20:08:39.856690 ip-10-0-143-108 systemd[1]: Started Kubernetes Kubelet. Jul 10 20:08:39.857876 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.857844 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 10 20:08:39.859986 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.859969 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 20:08:39.861221 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.860342 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-108.ec2.internal.18c1068a9ed8a0d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-108.ec2.internal,UID:ip-10-0-143-108.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-108.ec2.internal,},FirstTimestamp:2026-07-10 20:08:39.855677654 +0000 UTC m=+0.328646970,LastTimestamp:2026-07-10 20:08:39.855677654 +0000 UTC m=+0.328646970,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-108.ec2.internal,}" Jul 10 20:08:39.861465 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.861431 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 20:08:39.863342 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.863322 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 20:08:39.863441 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.863326 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 20:08:39.863534 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.863427 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:39.863878 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.863845 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 20:08:39.863973 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.863853 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 20:08:39.863973 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.863915 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 20:08:39.864080 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.864052 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 10 20:08:39.864080 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.864062 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 10 20:08:39.864356 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.864327 2565 factory.go:55] Registering systemd factory Jul 10 20:08:39.864356 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.864348 2565 factory.go:223] Registration of the systemd container factory successfully Jul 10 20:08:39.864581 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.864563 2565 factory.go:153] Registering CRI-O factory Jul 10 20:08:39.864635 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.864594 2565 factory.go:223] Registration of the crio container factory successfully Jul 10 20:08:39.864672 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.864647 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 20:08:39.864725 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.864677 2565 factory.go:103] Registering Raw factory Jul 10 20:08:39.864725 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.864704 2565 manager.go:1196] Started watching for new ooms in manager Jul 10 20:08:39.865165 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.865142 2565 manager.go:349] Starting recovery of all containers Jul 10 20:08:39.866132 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.866096 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tk6b2" Jul 10 20:08:39.873795 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.873608 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-108.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 20:08:39.873942 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.873900 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 20:08:39.874375 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.874349 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tk6b2" Jul 10 20:08:39.875336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.875297 2565 manager.go:354] Recovery completed Jul 10 20:08:39.881484 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.881464 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:08:39.883589 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.883567 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:08:39.883657 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.883609 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:08:39.883657 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.883624 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:08:39.884206 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.884187 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 20:08:39.884206 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.884198 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:08:39.884206 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.884203 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 20:08:39.884377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.884218 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 10 20:08:39.884914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.884895 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:08:39.884990 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.884929 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:08:39.884990 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.884943 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:08:39.885373 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.885359 2565 policy_none.go:49] "None policy: Start" Jul 10 20:08:39.885434 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.885376 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 20:08:39.885434 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.885409 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 20:08:39.886388 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.886376 2565 policy_none.go:47] "Start" Jul 10 20:08:39.926603 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.926578 2565 manager.go:337] "Starting Device Plugin manager" Jul 10 20:08:39.926731 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.926621 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 20:08:39.926731 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.926636 2565 server.go:87] "Starting device plugin registration server" Jul 10 20:08:39.927020 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.927005 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 20:08:39.927100 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.927020 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 20:08:39.927160 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.927091 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 20:08:39.927230 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.927214 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 20:08:39.927230 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:39.927228 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 20:08:39.927760 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.927740 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 20:08:39.927840 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:39.927787 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.027333 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.027285 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:08:40.028565 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.028545 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:08:40.028646 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.028584 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:08:40.028646 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.028596 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:08:40.028646 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.028629 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.032830 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.032798 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 20:08:40.034171 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.034147 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 20:08:40.034231 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.034198 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 20:08:40.034231 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.034223 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 20:08:40.034313 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.034264 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 20:08:40.036490 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.036352 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.036490 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.036376 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-108.ec2.internal\": node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.036649 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.036589 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:08:40.066282 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.066257 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.135024 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.134982 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal"] Jul 10 20:08:40.135150 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.135077 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:08:40.136690 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.136662 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:08:40.136804 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.136701 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:08:40.136804 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.136715 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:08:40.138118 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.138100 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:08:40.138234 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.138196 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.138234 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.138220 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:08:40.138816 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.138795 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:08:40.138924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.138824 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:08:40.138924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.138825 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:08:40.138924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.138834 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:08:40.138924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.138845 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:08:40.138924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.138854 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:08:40.139853 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.139839 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.139919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.139884 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 20:08:40.140523 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.140507 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientMemory" Jul 10 20:08:40.140609 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.140532 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 20:08:40.140609 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.140542 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeHasSufficientPID" Jul 10 20:08:40.163776 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.163748 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-108.ec2.internal\" not found" node="ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.166537 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.166513 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.168664 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.168648 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-108.ec2.internal\" not found" node="ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.265660 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.265622 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07aa0ca73f3d80d2650c468cb7b052ee-config\") pod \"kube-apiserver-proxy-ip-10-0-143-108.ec2.internal\" (UID: \"07aa0ca73f3d80d2650c468cb7b052ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.265660 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.265658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/634599de4fb1750b2ea625fb170f15df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal\" (UID: \"634599de4fb1750b2ea625fb170f15df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.265909 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.265680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/634599de4fb1750b2ea625fb170f15df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal\" (UID: \"634599de4fb1750b2ea625fb170f15df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.266701 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.266685 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.366430 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.366348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07aa0ca73f3d80d2650c468cb7b052ee-config\") pod \"kube-apiserver-proxy-ip-10-0-143-108.ec2.internal\" (UID: \"07aa0ca73f3d80d2650c468cb7b052ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.366430 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.366306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/07aa0ca73f3d80d2650c468cb7b052ee-config\") pod \"kube-apiserver-proxy-ip-10-0-143-108.ec2.internal\" (UID: \"07aa0ca73f3d80d2650c468cb7b052ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.366430 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.366428 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/634599de4fb1750b2ea625fb170f15df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal\" (UID: \"634599de4fb1750b2ea625fb170f15df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.366614 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.366410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/634599de4fb1750b2ea625fb170f15df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal\" (UID: \"634599de4fb1750b2ea625fb170f15df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.366614 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.366458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/634599de4fb1750b2ea625fb170f15df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal\" (UID: \"634599de4fb1750b2ea625fb170f15df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.366614 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.366479 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/634599de4fb1750b2ea625fb170f15df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal\" (UID: \"634599de4fb1750b2ea625fb170f15df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.367451 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.367430 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.466971 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.466922 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.467968 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.467940 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.471699 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.471374 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" Jul 10 20:08:40.568850 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.568804 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.670905 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.670781 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.771174 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.771126 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.807635 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.807599 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 20:08:40.808420 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.807785 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 20:08:40.871839 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.871803 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:40.879466 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.879432 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 20:03:40 +0000 UTC" deadline="2028-03-26 07:12:03.898087486 +0000 UTC" Jul 10 20:08:40.879466 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:40.879465 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14987h3m23.018628163s" Jul 10 20:08:40.972228 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:40.972189 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:41.052745 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.052716 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:08:41.054015 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:41.053983 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07aa0ca73f3d80d2650c468cb7b052ee.slice/crio-c42a40e617d45119af19d3219684397121b17be42b60e9c1a5e3e73cf3d0d753 WatchSource:0}: Error finding container c42a40e617d45119af19d3219684397121b17be42b60e9c1a5e3e73cf3d0d753: Status 404 returned error can't find the container with id c42a40e617d45119af19d3219684397121b17be42b60e9c1a5e3e73cf3d0d753 Jul 10 20:08:41.054570 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:41.054544 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634599de4fb1750b2ea625fb170f15df.slice/crio-04db8b4527e4e16bd45d1508bb98d45c53c3e193ec541102c20c1eea50e16632 WatchSource:0}: Error finding container 04db8b4527e4e16bd45d1508bb98d45c53c3e193ec541102c20c1eea50e16632: Status 404 returned error can't find the container with id 04db8b4527e4e16bd45d1508bb98d45c53c3e193ec541102c20c1eea50e16632 Jul 10 20:08:41.059095 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.059078 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:08:41.073208 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:41.073177 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:41.173799 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:41.173738 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:41.274961 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:41.274843 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-108.ec2.internal\" not found" Jul 10 20:08:41.294755 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.294734 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:08:41.302616 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.302583 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:08:41.364490 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.364451 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal" Jul 10 20:08:41.373115 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.373082 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 20:08:41.374083 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.374052 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" Jul 10 20:08:41.384043 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.384017 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 20:08:41.847337 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.847304 2565 apiserver.go:52] "Watching apiserver" Jul 10 20:08:41.854478 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.854438 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 20:08:41.855965 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.855919 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-m7gpw","kube-system/konnectivity-agent-l676f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd","openshift-image-registry/node-ca-5kkzz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal","openshift-multus/multus-additional-cni-plugins-wshtg","openshift-multus/multus-s9rr9","kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal","openshift-cluster-node-tuning-operator/tuned-q8kwg","openshift-dns/node-resolver-znc59","openshift-multus/network-metrics-daemon-xfmzc","openshift-network-diagnostics/network-check-target-d62wz","openshift-network-operator/iptables-alerter-wzwz8","openshift-ovn-kubernetes/ovnkube-node-v8bxg"] Jul 10 20:08:41.858065 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.857292 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.858395 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.858372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:41.858485 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:41.858456 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:08:41.859997 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.859516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.860484 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.860263 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 20:08:41.860484 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.860275 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 20:08:41.860484 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.860266 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 20:08:41.860689 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.860638 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.860749 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.860691 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 20:08:41.860749 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.860700 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-29ffz\"" Jul 10 20:08:41.861623 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.861600 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 20:08:41.862094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.862063 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 20:08:41.862686 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.862323 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-946ng\"" Jul 10 20:08:41.862686 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.862550 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 20:08:41.863090 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.863068 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 20:08:41.863326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.863308 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 20:08:41.863526 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.863510 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 20:08:41.863626 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.863509 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.863769 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.863755 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.864664 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.863312 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qlq7z\"" Jul 10 20:08:41.865304 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.865264 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 20:08:41.865304 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.865297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.865624 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.865601 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 20:08:41.867031 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.867012 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:41.868439 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.868413 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 20:08:41.868718 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.868696 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.869128 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r4dgd\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.869522 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.869618 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.869645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.869753 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.869790 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.870044 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cjp5w\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.870131 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.870048 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.870184 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.870519 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l676f" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.870601 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.871531 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.871593 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7mn2p\"" Jul 10 20:08:41.871811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.871757 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6hl5f\"" Jul 10 20:08:41.873607 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.873584 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:41.873712 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:41.873653 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:08:41.874507 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.874443 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 20:08:41.874611 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.874577 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 20:08:41.874611 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.874590 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 20:08:41.874611 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.874606 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t9458\"" Jul 10 20:08:41.875072 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.875054 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vtwfw\"" Jul 10 20:08:41.875139 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.875053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:41.875177 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:41.875135 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:08:41.875351 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.875325 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 20:08:41.875538 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.875473 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 20:08:41.877228 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.877207 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 20:08:41.878127 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878099 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-socket-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.878225 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-systemd-units\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.878225 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-run-ovn\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.878225 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-run-ovn-kubernetes\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.878374 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878251 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-sysctl-conf\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.878374 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-lib-modules\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.878374 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-run-netns\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.878374 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878368 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-run-systemd\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.878530 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878400 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-kubernetes\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.878530 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878425 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:41.878530 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-kubelet\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.878530 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878498 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fttgq\" (UniqueName: \"kubernetes.io/projected/95101adb-8412-4fd6-8e51-6668c43d5379-kube-api-access-fttgq\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.878530 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93addf78-13b4-48cc-bb04-f4134b872483-serviceca\") pod \"node-ca-5kkzz\" (UID: \"93addf78-13b4-48cc-bb04-f4134b872483\") " pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.878741 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878550 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01fdc1af-61b3-4a78-8b96-2235b760e26d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.878741 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-cni-netd\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.878741 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88851b04-9c40-4f05-a3f2-7d62c45e32db-env-overrides\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.878741 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88851b04-9c40-4f05-a3f2-7d62c45e32db-ovnkube-script-lib\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.878741 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878695 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-cnibin\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.878741 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-var-lib-cni-bin\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.879004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-run\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.879004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-var-lib-kubelet\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.879004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01fdc1af-61b3-4a78-8b96-2235b760e26d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.879004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-etc-openvswitch\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.879004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-sys\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.879004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878904 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-os-release\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.879004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93addf78-13b4-48cc-bb04-f4134b872483-host\") pod \"node-ca-5kkzz\" (UID: \"93addf78-13b4-48cc-bb04-f4134b872483\") " pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.879004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.878988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879021 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88tt\" (UniqueName: \"kubernetes.io/projected/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-kube-api-access-j88tt\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-node-log\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-modprobe-d\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95101adb-8412-4fd6-8e51-6668c43d5379-tmp\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pmpk\" (UniqueName: \"kubernetes.io/projected/3ac597d8-dc79-46ad-954e-bc34432d454a-kube-api-access-5pmpk\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-sys-fs\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-run-openvswitch\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-systemd\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879286 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-var-lib-kubelet\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879308 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95101adb-8412-4fd6-8e51-6668c43d5379-etc-tuned\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-hostroot\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-sysconfig\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.879399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879390 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-run-k8s-cni-cncf-io\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-var-lib-cni-multus\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-cni-bin\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88851b04-9c40-4f05-a3f2-7d62c45e32db-ovn-node-metrics-cert\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-os-release\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-registration-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-run-netns\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879590 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-conf-dir\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-system-cni-dir\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879644 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-var-lib-openvswitch\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-sysctl-d\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879692 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t489\" (UniqueName: \"kubernetes.io/projected/e2cf593d-020d-43fa-afd6-d85e2a98c2da-kube-api-access-2t489\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6fd\" (UniqueName: \"kubernetes.io/projected/01fdc1af-61b3-4a78-8b96-2235b760e26d-kube-api-access-ms6fd\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-slash\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879756 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-log-socket\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw98c\" (UniqueName: \"kubernetes.io/projected/88851b04-9c40-4f05-a3f2-7d62c45e32db-kube-api-access-cw98c\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.880082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879803 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-cni-dir\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01fdc1af-61b3-4a78-8b96-2235b760e26d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-device-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879952 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.879980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88851b04-9c40-4f05-a3f2-7d62c45e32db-ovnkube-config\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.880010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-host\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.880036 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-system-cni-dir\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.880060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vpc6\" (UniqueName: \"kubernetes.io/projected/93addf78-13b4-48cc-bb04-f4134b872483-kube-api-access-2vpc6\") pod \"node-ca-5kkzz\" (UID: \"93addf78-13b4-48cc-bb04-f4134b872483\") " pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.880085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-cnibin\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.880111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2cf593d-020d-43fa-afd6-d85e2a98c2da-cni-binary-copy\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.880138 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-socket-dir-parent\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.880163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-daemon-config\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.880213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-run-multus-certs\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.880586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.880235 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-etc-kubernetes\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.886089 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.886052 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 20:08:41.901096 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.901069 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vx552" Jul 10 20:08:41.910079 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.910053 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vx552" Jul 10 20:08:41.965102 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.965071 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 20:08:41.980514 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980467 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-slash\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.980514 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-log-socket\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw98c\" (UniqueName: \"kubernetes.io/projected/88851b04-9c40-4f05-a3f2-7d62c45e32db-kube-api-access-cw98c\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-cni-dir\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01fdc1af-61b3-4a78-8b96-2235b760e26d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980644 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-slash\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-device-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88851b04-9c40-4f05-a3f2-7d62c45e32db-ovnkube-config\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-host\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.980778 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-system-cni-dir\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-log-socket\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vpc6\" (UniqueName: \"kubernetes.io/projected/93addf78-13b4-48cc-bb04-f4134b872483-kube-api-access-2vpc6\") pod \"node-ca-5kkzz\" (UID: \"93addf78-13b4-48cc-bb04-f4134b872483\") " pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-cnibin\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd973a57-23d9-4ee5-985f-cae1defe06cf-hosts-file\") pod \"node-resolver-znc59\" (UID: \"dd973a57-23d9-4ee5-985f-cae1defe06cf\") " pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2cf593d-020d-43fa-afd6-d85e2a98c2da-cni-binary-copy\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-socket-dir-parent\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-daemon-config\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980965 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-run-multus-certs\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980983 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-etc-kubernetes\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e7a5331-206f-47f7-b6ee-237be41026bd-kubelet-config\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981021 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-socket-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-systemd-units\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-run-ovn\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-run-ovn-kubernetes\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-sysctl-conf\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-lib-modules\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.981277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-run-netns\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-run-systemd\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-kubernetes\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-kubelet\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fttgq\" (UniqueName: \"kubernetes.io/projected/95101adb-8412-4fd6-8e51-6668c43d5379-kube-api-access-fttgq\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93addf78-13b4-48cc-bb04-f4134b872483-serviceca\") pod \"node-ca-5kkzz\" (UID: \"93addf78-13b4-48cc-bb04-f4134b872483\") " pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01fdc1af-61b3-4a78-8b96-2235b760e26d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ef5ad9d-15d0-4ec4-bb64-106753d90471-host-slash\") pod \"iptables-alerter-wzwz8\" (UID: \"1ef5ad9d-15d0-4ec4-bb64-106753d90471\") " pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd973a57-23d9-4ee5-985f-cae1defe06cf-tmp-dir\") pod \"node-resolver-znc59\" (UID: \"dd973a57-23d9-4ee5-985f-cae1defe06cf\") " pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981310 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-cni-netd\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88851b04-9c40-4f05-a3f2-7d62c45e32db-env-overrides\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88851b04-9c40-4f05-a3f2-7d62c45e32db-ovnkube-script-lib\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-cnibin\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-var-lib-cni-bin\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981412 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24f5c\" (UniqueName: \"kubernetes.io/projected/dd973a57-23d9-4ee5-985f-cae1defe06cf-kube-api-access-24f5c\") pod \"node-resolver-znc59\" (UID: \"dd973a57-23d9-4ee5-985f-cae1defe06cf\") " pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:41.982168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e7a5331-206f-47f7-b6ee-237be41026bd-dbus\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981466 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-run\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-var-lib-kubelet\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981514 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01fdc1af-61b3-4a78-8b96-2235b760e26d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-etc-openvswitch\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-sys\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-os-release\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93addf78-13b4-48cc-bb04-f4134b872483-host\") pod \"node-ca-5kkzz\" (UID: \"93addf78-13b4-48cc-bb04-f4134b872483\") " pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981601 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88tt\" (UniqueName: \"kubernetes.io/projected/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-kube-api-access-j88tt\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.980745 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-cni-dir\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-host\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-socket-dir-parent\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-cnibin\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-system-cni-dir\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:41.981972 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:41.982970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.981992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-kubelet\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982081 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-sysctl-conf\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-node-log\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:41.982165 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs podName:3ac597d8-dc79-46ad-954e-bc34432d454a nodeName:}" failed. No retries permitted until 2026-07-10 20:08:42.48213935 +0000 UTC m=+2.955108661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs") pod "network-metrics-daemon-xfmzc" (UID: "3ac597d8-dc79-46ad-954e-bc34432d454a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982193 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-run\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-modprobe-d\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95101adb-8412-4fd6-8e51-6668c43d5379-tmp\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6k9p\" (UniqueName: \"kubernetes.io/projected/1ef5ad9d-15d0-4ec4-bb64-106753d90471-kube-api-access-m6k9p\") pod \"iptables-alerter-wzwz8\" (UID: \"1ef5ad9d-15d0-4ec4-bb64-106753d90471\") " pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pmpk\" (UniqueName: \"kubernetes.io/projected/3ac597d8-dc79-46ad-954e-bc34432d454a-kube-api-access-5pmpk\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982288 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-modprobe-d\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-sys-fs\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-run-systemd\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982383 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-run-openvswitch\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982405 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982424 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-var-lib-kubelet\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-device-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982514 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-daemon-config\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.983844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-run-openvswitch\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-systemd\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-systemd-units\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-var-lib-kubelet\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95101adb-8412-4fd6-8e51-6668c43d5379-etc-tuned\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982662 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93addf78-13b4-48cc-bb04-f4134b872483-serviceca\") pod \"node-ca-5kkzz\" (UID: \"93addf78-13b4-48cc-bb04-f4134b872483\") " pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982668 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-run-ovn\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-hostroot\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-run-ovn-kubernetes\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46f45b82-e1e3-4b93-aca9-bbb73ac320e2-agent-certs\") pod \"konnectivity-agent-l676f\" (UID: \"46f45b82-e1e3-4b93-aca9-bbb73ac320e2\") " pod="kube-system/konnectivity-agent-l676f" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982711 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-node-log\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982283 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-lib-modules\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-sysconfig\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982986 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01fdc1af-61b3-4a78-8b96-2235b760e26d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.982990 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-run-k8s-cni-cncf-io\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-run-k8s-cni-cncf-io\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983035 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-var-lib-cni-multus\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-cni-bin\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.984651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983075 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-sysconfig\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983081 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88851b04-9c40-4f05-a3f2-7d62c45e32db-env-overrides\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88851b04-9c40-4f05-a3f2-7d62c45e32db-ovn-node-metrics-cert\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-os-release\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-registration-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-run-netns\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88851b04-9c40-4f05-a3f2-7d62c45e32db-ovnkube-script-lib\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-conf-dir\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-system-cni-dir\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983685 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88851b04-9c40-4f05-a3f2-7d62c45e32db-ovnkube-config\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2cf593d-020d-43fa-afd6-d85e2a98c2da-cni-binary-copy\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-var-lib-openvswitch\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-sysctl-d\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t489\" (UniqueName: \"kubernetes.io/projected/e2cf593d-020d-43fa-afd6-d85e2a98c2da-kube-api-access-2t489\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983785 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-os-release\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-var-lib-openvswitch\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-registration-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.985516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-run-netns\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.983986 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01fdc1af-61b3-4a78-8b96-2235b760e26d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984035 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-multus-conf-dir\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-sysctl-d\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-sys\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984185 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01fdc1af-61b3-4a78-8b96-2235b760e26d-system-cni-dir\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984199 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-cni-bin\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984359 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-cni-netd\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6fd\" (UniqueName: \"kubernetes.io/projected/01fdc1af-61b3-4a78-8b96-2235b760e26d-kube-api-access-ms6fd\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984530 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01fdc1af-61b3-4a78-8b96-2235b760e26d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984689 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984702 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-etc-kubernetes\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984775 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-systemd\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984906 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-hostroot\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.984995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-etc-kubernetes\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985120 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-socket-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95101adb-8412-4fd6-8e51-6668c43d5379-var-lib-kubelet\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985516 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-run-multus-certs\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.986462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985580 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93addf78-13b4-48cc-bb04-f4134b872483-host\") pod \"node-ca-5kkzz\" (UID: \"93addf78-13b4-48cc-bb04-f4134b872483\") " pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985676 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-etc-openvswitch\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985723 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-cnibin\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-os-release\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-run-netns\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.985938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-var-lib-cni-multus\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.986052 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e2cf593d-020d-43fa-afd6-d85e2a98c2da-host-var-lib-cni-bin\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.986132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88851b04-9c40-4f05-a3f2-7d62c45e32db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.986178 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-sys-fs\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.986328 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ef5ad9d-15d0-4ec4-bb64-106753d90471-iptables-alerter-script\") pod \"iptables-alerter-wzwz8\" (UID: \"1ef5ad9d-15d0-4ec4-bb64-106753d90471\") " pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:41.987377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.986378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46f45b82-e1e3-4b93-aca9-bbb73ac320e2-konnectivity-ca\") pod \"konnectivity-agent-l676f\" (UID: \"46f45b82-e1e3-4b93-aca9-bbb73ac320e2\") " pod="kube-system/konnectivity-agent-l676f" Jul 10 20:08:41.989672 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.988704 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95101adb-8412-4fd6-8e51-6668c43d5379-tmp\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.989672 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.989462 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/95101adb-8412-4fd6-8e51-6668c43d5379-etc-tuned\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.991548 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.990009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88851b04-9c40-4f05-a3f2-7d62c45e32db-ovn-node-metrics-cert\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.992181 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.992026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vpc6\" (UniqueName: \"kubernetes.io/projected/93addf78-13b4-48cc-bb04-f4134b872483-kube-api-access-2vpc6\") pod \"node-ca-5kkzz\" (UID: \"93addf78-13b4-48cc-bb04-f4134b872483\") " pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:41.994634 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.994608 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw98c\" (UniqueName: \"kubernetes.io/projected/88851b04-9c40-4f05-a3f2-7d62c45e32db-kube-api-access-cw98c\") pod \"ovnkube-node-v8bxg\" (UID: \"88851b04-9c40-4f05-a3f2-7d62c45e32db\") " pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:41.994772 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.994687 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6fd\" (UniqueName: \"kubernetes.io/projected/01fdc1af-61b3-4a78-8b96-2235b760e26d-kube-api-access-ms6fd\") pod \"multus-additional-cni-plugins-wshtg\" (UID: \"01fdc1af-61b3-4a78-8b96-2235b760e26d\") " pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:41.995470 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.995430 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t489\" (UniqueName: \"kubernetes.io/projected/e2cf593d-020d-43fa-afd6-d85e2a98c2da-kube-api-access-2t489\") pod \"multus-s9rr9\" (UID: \"e2cf593d-020d-43fa-afd6-d85e2a98c2da\") " pod="openshift-multus/multus-s9rr9" Jul 10 20:08:41.996714 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.996692 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fttgq\" (UniqueName: \"kubernetes.io/projected/95101adb-8412-4fd6-8e51-6668c43d5379-kube-api-access-fttgq\") pod \"tuned-q8kwg\" (UID: \"95101adb-8412-4fd6-8e51-6668c43d5379\") " pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:41.997051 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.997019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88tt\" (UniqueName: \"kubernetes.io/projected/27ed50d7-0bc0-4b83-a635-fe9c168a06cc-kube-api-access-j88tt\") pod \"aws-ebs-csi-driver-node-5rcsd\" (UID: \"27ed50d7-0bc0-4b83-a635-fe9c168a06cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:41.998085 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:41.998061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pmpk\" (UniqueName: \"kubernetes.io/projected/3ac597d8-dc79-46ad-954e-bc34432d454a-kube-api-access-5pmpk\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:42.040428 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.040360 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" event={"ID":"634599de4fb1750b2ea625fb170f15df","Type":"ContainerStarted","Data":"04db8b4527e4e16bd45d1508bb98d45c53c3e193ec541102c20c1eea50e16632"} Jul 10 20:08:42.041769 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.041732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal" event={"ID":"07aa0ca73f3d80d2650c468cb7b052ee","Type":"ContainerStarted","Data":"c42a40e617d45119af19d3219684397121b17be42b60e9c1a5e3e73cf3d0d753"} Jul 10 20:08:42.087379 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ef5ad9d-15d0-4ec4-bb64-106753d90471-iptables-alerter-script\") pod \"iptables-alerter-wzwz8\" (UID: \"1ef5ad9d-15d0-4ec4-bb64-106753d90471\") " pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:42.087379 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087391 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46f45b82-e1e3-4b93-aca9-bbb73ac320e2-konnectivity-ca\") pod \"konnectivity-agent-l676f\" (UID: \"46f45b82-e1e3-4b93-aca9-bbb73ac320e2\") " pod="kube-system/konnectivity-agent-l676f" Jul 10 20:08:42.087637 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd973a57-23d9-4ee5-985f-cae1defe06cf-hosts-file\") pod \"node-resolver-znc59\" (UID: \"dd973a57-23d9-4ee5-985f-cae1defe06cf\") " pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:42.087637 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e7a5331-206f-47f7-b6ee-237be41026bd-kubelet-config\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:42.087637 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087514 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ef5ad9d-15d0-4ec4-bb64-106753d90471-host-slash\") pod \"iptables-alerter-wzwz8\" (UID: \"1ef5ad9d-15d0-4ec4-bb64-106753d90471\") " pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:42.087637 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd973a57-23d9-4ee5-985f-cae1defe06cf-hosts-file\") pod \"node-resolver-znc59\" (UID: \"dd973a57-23d9-4ee5-985f-cae1defe06cf\") " pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:42.087637 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5e7a5331-206f-47f7-b6ee-237be41026bd-kubelet-config\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:42.087914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087641 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ef5ad9d-15d0-4ec4-bb64-106753d90471-host-slash\") pod \"iptables-alerter-wzwz8\" (UID: \"1ef5ad9d-15d0-4ec4-bb64-106753d90471\") " pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:42.087914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087677 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd973a57-23d9-4ee5-985f-cae1defe06cf-tmp-dir\") pod \"node-resolver-znc59\" (UID: \"dd973a57-23d9-4ee5-985f-cae1defe06cf\") " pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:42.087914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:42.087914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24f5c\" (UniqueName: \"kubernetes.io/projected/dd973a57-23d9-4ee5-985f-cae1defe06cf-kube-api-access-24f5c\") pod \"node-resolver-znc59\" (UID: \"dd973a57-23d9-4ee5-985f-cae1defe06cf\") " pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:42.087914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e7a5331-206f-47f7-b6ee-237be41026bd-dbus\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:42.087914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:42.087914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6k9p\" (UniqueName: \"kubernetes.io/projected/1ef5ad9d-15d0-4ec4-bb64-106753d90471-kube-api-access-m6k9p\") pod \"iptables-alerter-wzwz8\" (UID: \"1ef5ad9d-15d0-4ec4-bb64-106753d90471\") " pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:42.088275 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.087863 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46f45b82-e1e3-4b93-aca9-bbb73ac320e2-agent-certs\") pod \"konnectivity-agent-l676f\" (UID: \"46f45b82-e1e3-4b93-aca9-bbb73ac320e2\") " pod="kube-system/konnectivity-agent-l676f" Jul 10 20:08:42.088275 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.088047 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd973a57-23d9-4ee5-985f-cae1defe06cf-tmp-dir\") pod \"node-resolver-znc59\" (UID: \"dd973a57-23d9-4ee5-985f-cae1defe06cf\") " pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:42.088275 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.088148 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46f45b82-e1e3-4b93-aca9-bbb73ac320e2-konnectivity-ca\") pod \"konnectivity-agent-l676f\" (UID: \"46f45b82-e1e3-4b93-aca9-bbb73ac320e2\") " pod="kube-system/konnectivity-agent-l676f" Jul 10 20:08:42.088275 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.088178 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:42.088275 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.088256 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret podName:5e7a5331-206f-47f7-b6ee-237be41026bd nodeName:}" failed. No retries permitted until 2026-07-10 20:08:42.588237134 +0000 UTC m=+3.061206444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret") pod "global-pull-secret-syncer-m7gpw" (UID: "5e7a5331-206f-47f7-b6ee-237be41026bd") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:42.088275 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.088177 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ef5ad9d-15d0-4ec4-bb64-106753d90471-iptables-alerter-script\") pod \"iptables-alerter-wzwz8\" (UID: \"1ef5ad9d-15d0-4ec4-bb64-106753d90471\") " pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:42.088613 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.088328 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5e7a5331-206f-47f7-b6ee-237be41026bd-dbus\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:42.091060 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.091004 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46f45b82-e1e3-4b93-aca9-bbb73ac320e2-agent-certs\") pod \"konnectivity-agent-l676f\" (UID: \"46f45b82-e1e3-4b93-aca9-bbb73ac320e2\") " pod="kube-system/konnectivity-agent-l676f" Jul 10 20:08:42.097427 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.097356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6k9p\" (UniqueName: \"kubernetes.io/projected/1ef5ad9d-15d0-4ec4-bb64-106753d90471-kube-api-access-m6k9p\") pod \"iptables-alerter-wzwz8\" (UID: \"1ef5ad9d-15d0-4ec4-bb64-106753d90471\") " pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:42.098204 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.097860 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:08:42.098204 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.097912 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:08:42.098204 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.097927 2565 projected.go:196] Error preparing data for projected volume kube-api-access-twkll for pod openshift-network-diagnostics/network-check-target-d62wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:42.098204 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.098002 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll podName:4e7ce493-89e1-4d2d-ae2f-c57f227d2e26 nodeName:}" failed. No retries permitted until 2026-07-10 20:08:42.597982197 +0000 UTC m=+3.070951528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-twkll" (UniqueName: "kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll") pod "network-check-target-d62wz" (UID: "4e7ce493-89e1-4d2d-ae2f-c57f227d2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:42.100789 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.100761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24f5c\" (UniqueName: \"kubernetes.io/projected/dd973a57-23d9-4ee5-985f-cae1defe06cf-kube-api-access-24f5c\") pod \"node-resolver-znc59\" (UID: \"dd973a57-23d9-4ee5-985f-cae1defe06cf\") " pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:42.171765 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.171714 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s9rr9" Jul 10 20:08:42.181960 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.181932 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" Jul 10 20:08:42.191055 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.191023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5kkzz" Jul 10 20:08:42.204135 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.204095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:08:42.213135 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.213111 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wshtg" Jul 10 20:08:42.222045 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.222021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" Jul 10 20:08:42.230826 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.230806 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wzwz8" Jul 10 20:08:42.245789 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.245757 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l676f" Jul 10 20:08:42.252644 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.252619 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-znc59" Jul 10 20:08:42.492351 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.492130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:42.492533 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.492326 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:42.492533 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.492475 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs podName:3ac597d8-dc79-46ad-954e-bc34432d454a nodeName:}" failed. No retries permitted until 2026-07-10 20:08:43.492453358 +0000 UTC m=+3.965422665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs") pod "network-metrics-daemon-xfmzc" (UID: "3ac597d8-dc79-46ad-954e-bc34432d454a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:42.592744 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.592693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:42.592985 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.592846 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:42.592985 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.592922 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret podName:5e7a5331-206f-47f7-b6ee-237be41026bd nodeName:}" failed. No retries permitted until 2026-07-10 20:08:43.592892324 +0000 UTC m=+4.065861644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret") pod "global-pull-secret-syncer-m7gpw" (UID: "5e7a5331-206f-47f7-b6ee-237be41026bd") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:42.693522 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.693409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:42.693669 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.693635 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:08:42.693722 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.693675 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:08:42.693722 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.693691 2565 projected.go:196] Error preparing data for projected volume kube-api-access-twkll for pod openshift-network-diagnostics/network-check-target-d62wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:42.693793 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:42.693776 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll podName:4e7ce493-89e1-4d2d-ae2f-c57f227d2e26 nodeName:}" failed. No retries permitted until 2026-07-10 20:08:43.693740903 +0000 UTC m=+4.166710231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-twkll" (UniqueName: "kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll") pod "network-check-target-d62wz" (UID: "4e7ce493-89e1-4d2d-ae2f-c57f227d2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:42.912876 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.912800 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 20:03:42 +0000 UTC" deadline="2028-04-06 21:21:47.67344573 +0000 UTC" Jul 10 20:08:42.912876 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:42.912882 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15265h13m4.760583475s" Jul 10 20:08:43.166122 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:43.166093 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cf593d_020d_43fa_afd6_d85e2a98c2da.slice/crio-2a4242970bcec092869a9c19d005d26f3be3c8ff3c5bfbb55a6587a360ac9d8a WatchSource:0}: Error finding container 2a4242970bcec092869a9c19d005d26f3be3c8ff3c5bfbb55a6587a360ac9d8a: Status 404 returned error can't find the container with id 2a4242970bcec092869a9c19d005d26f3be3c8ff3c5bfbb55a6587a360ac9d8a Jul 10 20:08:43.180346 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:43.180315 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93addf78_13b4_48cc_bb04_f4134b872483.slice/crio-813d2e20a1bbca5d2d1fca5561ba7c15b1dfdccfb02ce8b90e302666c584045f WatchSource:0}: Error finding container 813d2e20a1bbca5d2d1fca5561ba7c15b1dfdccfb02ce8b90e302666c584045f: Status 404 returned error can't find the container with id 813d2e20a1bbca5d2d1fca5561ba7c15b1dfdccfb02ce8b90e302666c584045f Jul 10 20:08:43.181073 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:43.181024 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88851b04_9c40_4f05_a3f2_7d62c45e32db.slice/crio-b671f1352960d9e03df1386e617c059393e81103e6dd14fd8088d341d0532894 WatchSource:0}: Error finding container b671f1352960d9e03df1386e617c059393e81103e6dd14fd8088d341d0532894: Status 404 returned error can't find the container with id b671f1352960d9e03df1386e617c059393e81103e6dd14fd8088d341d0532894 Jul 10 20:08:43.183194 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:43.183151 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ed50d7_0bc0_4b83_a635_fe9c168a06cc.slice/crio-6f5661d2d25b9aeb4fbb60a5148e22c59c98c1fce1a19ab2b28e695ecd7c64f2 WatchSource:0}: Error finding container 6f5661d2d25b9aeb4fbb60a5148e22c59c98c1fce1a19ab2b28e695ecd7c64f2: Status 404 returned error can't find the container with id 6f5661d2d25b9aeb4fbb60a5148e22c59c98c1fce1a19ab2b28e695ecd7c64f2 Jul 10 20:08:43.183490 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:43.183463 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd973a57_23d9_4ee5_985f_cae1defe06cf.slice/crio-d73d31c0b0d721dcb96108a47c0580257ce373941be96bdaaaa41b85a42e6d5a WatchSource:0}: Error finding container d73d31c0b0d721dcb96108a47c0580257ce373941be96bdaaaa41b85a42e6d5a: Status 404 returned error can't find the container with id d73d31c0b0d721dcb96108a47c0580257ce373941be96bdaaaa41b85a42e6d5a Jul 10 20:08:43.184685 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:43.184622 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95101adb_8412_4fd6_8e51_6668c43d5379.slice/crio-f0ff0505ab0244101dee86184cf9cf9f99a5ada5015e225efbebe07f10a9d948 WatchSource:0}: Error finding container f0ff0505ab0244101dee86184cf9cf9f99a5ada5015e225efbebe07f10a9d948: Status 404 returned error can't find the container with id f0ff0505ab0244101dee86184cf9cf9f99a5ada5015e225efbebe07f10a9d948 Jul 10 20:08:43.190523 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:08:43.189735 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef5ad9d_15d0_4ec4_bb64_106753d90471.slice/crio-c9e82ff455f5b44bbc89f8f819995ff7cb402b0b3d3c177ba67f6c28bf7d580a WatchSource:0}: Error finding container c9e82ff455f5b44bbc89f8f819995ff7cb402b0b3d3c177ba67f6c28bf7d580a: Status 404 returned error can't find the container with id c9e82ff455f5b44bbc89f8f819995ff7cb402b0b3d3c177ba67f6c28bf7d580a Jul 10 20:08:43.500704 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:43.500665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:43.500867 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:43.500847 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:43.500960 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:43.500947 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs podName:3ac597d8-dc79-46ad-954e-bc34432d454a nodeName:}" failed. No retries permitted until 2026-07-10 20:08:45.500924237 +0000 UTC m=+5.973893543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs") pod "network-metrics-daemon-xfmzc" (UID: "3ac597d8-dc79-46ad-954e-bc34432d454a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:43.602128 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:43.602078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:43.602313 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:43.602238 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:43.602411 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:43.602321 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret podName:5e7a5331-206f-47f7-b6ee-237be41026bd nodeName:}" failed. No retries permitted until 2026-07-10 20:08:45.602300988 +0000 UTC m=+6.075270300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret") pod "global-pull-secret-syncer-m7gpw" (UID: "5e7a5331-206f-47f7-b6ee-237be41026bd") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:43.704111 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:43.703256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:43.704111 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:43.703486 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:08:43.704111 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:43.703507 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:08:43.704111 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:43.703521 2565 projected.go:196] Error preparing data for projected volume kube-api-access-twkll for pod openshift-network-diagnostics/network-check-target-d62wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:43.704111 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:43.703594 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll podName:4e7ce493-89e1-4d2d-ae2f-c57f227d2e26 nodeName:}" failed. No retries permitted until 2026-07-10 20:08:45.703574451 +0000 UTC m=+6.176543778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-twkll" (UniqueName: "kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll") pod "network-check-target-d62wz" (UID: "4e7ce493-89e1-4d2d-ae2f-c57f227d2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:43.913111 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:43.913058 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 20:03:42 +0000 UTC" deadline="2028-01-21 10:44:29.411887312 +0000 UTC" Jul 10 20:08:43.913111 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:43.913108 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13430h35m45.4987846s" Jul 10 20:08:44.035157 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.035070 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:44.035341 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:44.035242 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:08:44.035819 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.035649 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:44.035819 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:44.035760 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:08:44.036301 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.036139 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:44.036301 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:44.036256 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:08:44.067043 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.066998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal" event={"ID":"07aa0ca73f3d80d2650c468cb7b052ee","Type":"ContainerStarted","Data":"2d09c8a8a961ee3119d01673a5d8fdc5e17b2de6a65d660dcaaa2d7a4cc1fb40"} Jul 10 20:08:44.072653 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.072615 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" event={"ID":"95101adb-8412-4fd6-8e51-6668c43d5379","Type":"ContainerStarted","Data":"f0ff0505ab0244101dee86184cf9cf9f99a5ada5015e225efbebe07f10a9d948"} Jul 10 20:08:44.080859 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.080777 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" event={"ID":"27ed50d7-0bc0-4b83-a635-fe9c168a06cc","Type":"ContainerStarted","Data":"6f5661d2d25b9aeb4fbb60a5148e22c59c98c1fce1a19ab2b28e695ecd7c64f2"} Jul 10 20:08:44.083421 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.083358 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-108.ec2.internal" podStartSLOduration=3.083337602 podStartE2EDuration="3.083337602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:08:44.082836347 +0000 UTC m=+4.555805677" watchObservedRunningTime="2026-07-10 20:08:44.083337602 +0000 UTC m=+4.556306933" Jul 10 20:08:44.087745 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.087709 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wzwz8" event={"ID":"1ef5ad9d-15d0-4ec4-bb64-106753d90471","Type":"ContainerStarted","Data":"c9e82ff455f5b44bbc89f8f819995ff7cb402b0b3d3c177ba67f6c28bf7d580a"} Jul 10 20:08:44.098553 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.098510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5kkzz" event={"ID":"93addf78-13b4-48cc-bb04-f4134b872483","Type":"ContainerStarted","Data":"813d2e20a1bbca5d2d1fca5561ba7c15b1dfdccfb02ce8b90e302666c584045f"} Jul 10 20:08:44.103747 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.103670 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wshtg" event={"ID":"01fdc1af-61b3-4a78-8b96-2235b760e26d","Type":"ContainerStarted","Data":"5ae846d9fd73a62292e00dd1ccb70c042584509c1bcad89a8741d6c5a7f8dfb7"} Jul 10 20:08:44.106054 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.105975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l676f" event={"ID":"46f45b82-e1e3-4b93-aca9-bbb73ac320e2","Type":"ContainerStarted","Data":"86c7dbf324ff025cc9bcf53dd7dd2207e33e6de4789559607118ccdb286ddddf"} Jul 10 20:08:44.119662 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.119583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-znc59" event={"ID":"dd973a57-23d9-4ee5-985f-cae1defe06cf","Type":"ContainerStarted","Data":"d73d31c0b0d721dcb96108a47c0580257ce373941be96bdaaaa41b85a42e6d5a"} Jul 10 20:08:44.126985 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.126811 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerStarted","Data":"b671f1352960d9e03df1386e617c059393e81103e6dd14fd8088d341d0532894"} Jul 10 20:08:44.134121 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.134057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s9rr9" event={"ID":"e2cf593d-020d-43fa-afd6-d85e2a98c2da","Type":"ContainerStarted","Data":"2a4242970bcec092869a9c19d005d26f3be3c8ff3c5bfbb55a6587a360ac9d8a"} Jul 10 20:08:44.160012 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.159598 2565 generic.go:356] "Generic (PLEG): container finished" podID="634599de4fb1750b2ea625fb170f15df" containerID="7e2be7d818c37469280f5d9e49cddfbd969f1aa7f855b13958f64e2088f8d2b1" exitCode=0 Jul 10 20:08:44.160012 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:44.159656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" event={"ID":"634599de4fb1750b2ea625fb170f15df","Type":"ContainerDied","Data":"7e2be7d818c37469280f5d9e49cddfbd969f1aa7f855b13958f64e2088f8d2b1"} Jul 10 20:08:45.180763 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:45.180723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" event={"ID":"634599de4fb1750b2ea625fb170f15df","Type":"ContainerStarted","Data":"ebc3b148b6490db5dcf8a48e92e62fffbb3fa78a6a80417a5dca998811739345"} Jul 10 20:08:45.520111 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:45.519388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:45.520111 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:45.519599 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:45.520111 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:45.519666 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs podName:3ac597d8-dc79-46ad-954e-bc34432d454a nodeName:}" failed. No retries permitted until 2026-07-10 20:08:49.519647729 +0000 UTC m=+9.992617050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs") pod "network-metrics-daemon-xfmzc" (UID: "3ac597d8-dc79-46ad-954e-bc34432d454a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:45.620394 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:45.620340 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:45.620603 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:45.620493 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:45.620603 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:45.620562 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret podName:5e7a5331-206f-47f7-b6ee-237be41026bd nodeName:}" failed. No retries permitted until 2026-07-10 20:08:49.620541409 +0000 UTC m=+10.093510731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret") pod "global-pull-secret-syncer-m7gpw" (UID: "5e7a5331-206f-47f7-b6ee-237be41026bd") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:45.722438 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:45.721687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:45.722438 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:45.721926 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:08:45.722438 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:45.721950 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:08:45.722438 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:45.721963 2565 projected.go:196] Error preparing data for projected volume kube-api-access-twkll for pod openshift-network-diagnostics/network-check-target-d62wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:45.722438 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:45.722029 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll podName:4e7ce493-89e1-4d2d-ae2f-c57f227d2e26 nodeName:}" failed. No retries permitted until 2026-07-10 20:08:49.722008511 +0000 UTC m=+10.194977820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-twkll" (UniqueName: "kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll") pod "network-check-target-d62wz" (UID: "4e7ce493-89e1-4d2d-ae2f-c57f227d2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:46.037529 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:46.037476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:46.037736 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:46.037617 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:08:46.038099 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:46.038064 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:46.038210 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:46.038174 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:08:46.038274 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:46.038240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:46.038353 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:46.038324 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:08:48.036014 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:48.035981 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:48.036014 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:48.036014 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:48.036592 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:48.036109 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:48.036592 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:48.036124 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:08:48.036592 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:48.036231 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:08:48.036592 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:48.036284 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:08:49.556408 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:49.556285 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:49.556904 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:49.556444 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:49.556904 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:49.556511 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs podName:3ac597d8-dc79-46ad-954e-bc34432d454a nodeName:}" failed. No retries permitted until 2026-07-10 20:08:57.556490183 +0000 UTC m=+18.029459493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs") pod "network-metrics-daemon-xfmzc" (UID: "3ac597d8-dc79-46ad-954e-bc34432d454a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:49.656960 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:49.656929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:49.657167 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:49.657066 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:49.657167 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:49.657133 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret podName:5e7a5331-206f-47f7-b6ee-237be41026bd nodeName:}" failed. No retries permitted until 2026-07-10 20:08:57.657113403 +0000 UTC m=+18.130082711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret") pod "global-pull-secret-syncer-m7gpw" (UID: "5e7a5331-206f-47f7-b6ee-237be41026bd") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:49.758079 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:49.758043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:49.758270 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:49.758248 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:08:49.758344 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:49.758310 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:08:49.758344 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:49.758326 2565 projected.go:196] Error preparing data for projected volume kube-api-access-twkll for pod openshift-network-diagnostics/network-check-target-d62wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:49.758445 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:49.758389 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll podName:4e7ce493-89e1-4d2d-ae2f-c57f227d2e26 nodeName:}" failed. No retries permitted until 2026-07-10 20:08:57.758370644 +0000 UTC m=+18.231339969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-twkll" (UniqueName: "kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll") pod "network-check-target-d62wz" (UID: "4e7ce493-89e1-4d2d-ae2f-c57f227d2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:50.037057 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:50.036970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:50.037057 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:50.037015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:50.037277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:50.036973 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:50.037277 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:50.037112 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:08:50.037382 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:50.037276 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:08:50.037433 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:50.037393 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:08:52.036462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:52.035522 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:52.036462 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:52.035669 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:08:52.036462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:52.036113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:52.036462 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:52.036204 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:08:52.036462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:52.036276 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:52.036462 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:52.036360 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:08:54.035429 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:54.035393 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:54.035944 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:54.035598 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:08:54.035944 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:54.035403 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:54.035944 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:54.035709 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:08:54.035944 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:54.035393 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:54.035944 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:54.035803 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:08:56.035171 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:56.035134 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:56.035605 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:56.035134 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:56.035605 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:56.035277 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:56.035605 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:56.035278 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:08:56.035605 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:56.035373 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:08:56.035605 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:56.035460 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:08:57.621150 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:57.621107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:57.621613 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:57.621294 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:57.621613 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:57.621378 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs podName:3ac597d8-dc79-46ad-954e-bc34432d454a nodeName:}" failed. No retries permitted until 2026-07-10 20:09:13.621356267 +0000 UTC m=+34.094325573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs") pod "network-metrics-daemon-xfmzc" (UID: "3ac597d8-dc79-46ad-954e-bc34432d454a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:08:57.722426 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:57.722380 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:57.722601 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:57.722560 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:57.722660 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:57.722637 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret podName:5e7a5331-206f-47f7-b6ee-237be41026bd nodeName:}" failed. No retries permitted until 2026-07-10 20:09:13.722614791 +0000 UTC m=+34.195584101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret") pod "global-pull-secret-syncer-m7gpw" (UID: "5e7a5331-206f-47f7-b6ee-237be41026bd") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:08:57.823020 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:57.822983 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:57.823159 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:57.823143 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:08:57.823211 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:57.823164 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:08:57.823211 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:57.823174 2565 projected.go:196] Error preparing data for projected volume kube-api-access-twkll for pod openshift-network-diagnostics/network-check-target-d62wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:57.823270 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:57.823229 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll podName:4e7ce493-89e1-4d2d-ae2f-c57f227d2e26 nodeName:}" failed. No retries permitted until 2026-07-10 20:09:13.823211077 +0000 UTC m=+34.296180400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-twkll" (UniqueName: "kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll") pod "network-check-target-d62wz" (UID: "4e7ce493-89e1-4d2d-ae2f-c57f227d2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:08:58.034886 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:58.034786 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:08:58.034886 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:58.034795 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:08:58.035104 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:08:58.034932 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:08:58.035104 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:58.034946 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:08:58.035104 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:58.035029 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:08:58.035250 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:08:58.035131 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:00.038256 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:00.037798 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:00.038256 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:00.037812 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:00.038256 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:00.037815 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:00.038256 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:00.037943 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:00.038256 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:00.038064 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:00.038256 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:00.038200 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:02.035483 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:02.035442 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:02.036129 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:02.035573 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:02.036129 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:02.035650 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:02.036129 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:02.035684 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:02.036129 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:02.035754 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:02.036129 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:02.035902 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:03.220976 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.220690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" event={"ID":"95101adb-8412-4fd6-8e51-6668c43d5379","Type":"ContainerStarted","Data":"172af7b222956b95af451f5522c92276ecc4dc0cbc794c1c1a8df6e311f219d5"} Jul 10 20:09:03.222273 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.222243 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" event={"ID":"27ed50d7-0bc0-4b83-a635-fe9c168a06cc","Type":"ContainerStarted","Data":"b018bcbbdd1053b643e517190eaac16ea000a7490e94e26cdc09df7e7e11cfe2"} Jul 10 20:09:03.223499 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.223478 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5kkzz" event={"ID":"93addf78-13b4-48cc-bb04-f4134b872483","Type":"ContainerStarted","Data":"79ec4109e09f8efb0305b16b9795c647e7ae227f9b016b2fef9bf264cff20495"} Jul 10 20:09:03.225115 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.224948 2565 generic.go:356] "Generic (PLEG): container finished" podID="01fdc1af-61b3-4a78-8b96-2235b760e26d" containerID="44094737230e3977120d446c949e61d04f10143a4690f953dac9fb692f90ffc9" exitCode=0 Jul 10 20:09:03.225212 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.225026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wshtg" event={"ID":"01fdc1af-61b3-4a78-8b96-2235b760e26d","Type":"ContainerDied","Data":"44094737230e3977120d446c949e61d04f10143a4690f953dac9fb692f90ffc9"} Jul 10 20:09:03.226663 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.226628 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l676f" event={"ID":"46f45b82-e1e3-4b93-aca9-bbb73ac320e2","Type":"ContainerStarted","Data":"700c5b39ff6a078c0642b349fd467eb45f3b5c41d46401c2339fd85556f971fb"} Jul 10 20:09:03.228372 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.228344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-znc59" event={"ID":"dd973a57-23d9-4ee5-985f-cae1defe06cf","Type":"ContainerStarted","Data":"32927a5b49d9ccbaea210420da791cdeaadf58c700037ac63b7b0caaa335d088"} Jul 10 20:09:03.230541 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.230519 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:09:03.230909 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.230886 2565 generic.go:356] "Generic (PLEG): container finished" podID="88851b04-9c40-4f05-a3f2-7d62c45e32db" containerID="4c4bf9e6f2306c7f79dd945f4f9c56bff4e0f979856fd2f385e3b148f6296b24" exitCode=1 Jul 10 20:09:03.230989 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.230965 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerStarted","Data":"3092a4dd19d5b0ba56c5bc81177e65de241b8888ec2e90e1a34a5a59bb12a84b"} Jul 10 20:09:03.231070 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.231055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerStarted","Data":"5cb90e246b0ee5e013bad665742dd316cca65e3c6384f3085b3e9aa8faa61c94"} Jul 10 20:09:03.231130 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.231113 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerDied","Data":"4c4bf9e6f2306c7f79dd945f4f9c56bff4e0f979856fd2f385e3b148f6296b24"} Jul 10 20:09:03.231178 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.231167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerStarted","Data":"b0f0e89e6022e52d60562a9b8e4739ad9b832f924c3fcee3eb9214afb0e9fb5f"} Jul 10 20:09:03.232444 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.232421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s9rr9" event={"ID":"e2cf593d-020d-43fa-afd6-d85e2a98c2da","Type":"ContainerStarted","Data":"b445662a4c87578838786cfc93243df11c7ca2b9969e0fb6dc727ad1a6236659"} Jul 10 20:09:03.238675 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.238624 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-108.ec2.internal" podStartSLOduration=22.238605959 podStartE2EDuration="22.238605959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:08:45.197462433 +0000 UTC m=+5.670431767" watchObservedRunningTime="2026-07-10 20:09:03.238605959 +0000 UTC m=+23.711575287" Jul 10 20:09:03.238973 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.238941 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q8kwg" podStartSLOduration=3.824473602 podStartE2EDuration="23.23892973s" totalImagesPullingTime="19.414456128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:03.238426291 +0000 UTC m=+23.711395617" watchObservedRunningTime="2026-07-10 20:09:03.23892973 +0000 UTC m=+23.711899056" Jul 10 20:09:03.256361 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.256312 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s9rr9" podStartSLOduration=3.803641886 podStartE2EDuration="23.256301222s" totalImagesPullingTime="19.452659336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:03.255737492 +0000 UTC m=+23.728706818" watchObservedRunningTime="2026-07-10 20:09:03.256301222 +0000 UTC m=+23.729270547" Jul 10 20:09:03.295014 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.294966 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-znc59" podStartSLOduration=3.881600771 podStartE2EDuration="23.294948922s" totalImagesPullingTime="19.413348151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:03.294852748 +0000 UTC m=+23.767822074" watchObservedRunningTime="2026-07-10 20:09:03.294948922 +0000 UTC m=+23.767918261" Jul 10 20:09:03.309381 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.309313 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5kkzz" podStartSLOduration=3.91515528 podStartE2EDuration="23.309289868s" totalImagesPullingTime="19.394134588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:03.308751594 +0000 UTC m=+23.781720920" watchObservedRunningTime="2026-07-10 20:09:03.309289868 +0000 UTC m=+23.782259195" Jul 10 20:09:03.325132 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.325072 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l676f" podStartSLOduration=12.410942547 podStartE2EDuration="23.325048541s" totalImagesPullingTime="10.914105994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:03.324889135 +0000 UTC m=+23.797858454" watchObservedRunningTime="2026-07-10 20:09:03.325048541 +0000 UTC m=+23.798017871" Jul 10 20:09:03.856729 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.856698 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 20:09:03.940310 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.940112 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T20:09:03.856721235Z","UUID":"bff90b73-8f63-4f9e-9003-4a856c056b1e","Handler":null,"Name":"","Endpoint":""} Jul 10 20:09:03.941884 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.941832 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 20:09:03.942033 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:03.941907 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 20:09:04.034679 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:04.034648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:04.034879 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:04.034646 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:04.034879 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:04.034771 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:04.034879 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:04.034650 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:04.034879 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:04.034829 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:04.035081 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:04.034943 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:04.237341 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:04.237244 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" event={"ID":"27ed50d7-0bc0-4b83-a635-fe9c168a06cc","Type":"ContainerStarted","Data":"a916ad2d913e4e6587cbc2073e42e68beaba57100dc3dd72b0640857a324c649"} Jul 10 20:09:04.238936 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:04.238903 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wzwz8" event={"ID":"1ef5ad9d-15d0-4ec4-bb64-106753d90471","Type":"ContainerStarted","Data":"fe4af824caa984cb0ae4dd6cac190e4ae18b0d90f962266e024a476ba332dfe9"} Jul 10 20:09:04.242192 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:04.242169 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:09:04.242664 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:04.242632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerStarted","Data":"e2c90958633be003e69022ee4fa45a63c6787115b8846ca88895cc9669cb6410"} Jul 10 20:09:04.242802 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:04.242723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerStarted","Data":"a600eece5f70ae988946dc8070621a4009ce9c21d9957964cfd250eade676d06"} Jul 10 20:09:04.253884 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:04.253809 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wzwz8" podStartSLOduration=4.844878379 podStartE2EDuration="24.253788007s" totalImagesPullingTime="19.408909628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:04.253280673 +0000 UTC m=+24.726250000" watchObservedRunningTime="2026-07-10 20:09:04.253788007 +0000 UTC m=+24.726757333" Jul 10 20:09:05.247159 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:05.246970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" event={"ID":"27ed50d7-0bc0-4b83-a635-fe9c168a06cc","Type":"ContainerStarted","Data":"ccab072e9749069bc31929ab60008e22d112f7872c3ca2c9d1db88f9424b2fb9"} Jul 10 20:09:05.265712 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:05.264880 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5rcsd" podStartSLOduration=4.139958414 podStartE2EDuration="25.264841482s" totalImagesPullingTime="21.124883068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:05.263680285 +0000 UTC m=+25.736649633" watchObservedRunningTime="2026-07-10 20:09:05.264841482 +0000 UTC m=+25.737810810" Jul 10 20:09:06.035849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:06.035354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:06.035849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:06.035354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:06.035849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:06.035505 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:06.035849 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:06.035489 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:06.035849 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:06.035599 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:06.035849 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:06.035673 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:06.252997 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:06.252957 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:09:06.253495 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:06.253418 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerStarted","Data":"701eb0de66f9f01079fff5721004023292f12a30de21473f4c924718b17deaee"} Jul 10 20:09:08.035703 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.035434 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:08.036393 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.035489 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:08.036393 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.035539 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:08.036393 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:08.035822 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:08.036393 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:08.035855 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:08.036393 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:08.035938 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:08.075237 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.075155 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l676f" Jul 10 20:09:08.075751 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.075734 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l676f" Jul 10 20:09:08.260006 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.259969 2565 generic.go:356] "Generic (PLEG): container finished" podID="01fdc1af-61b3-4a78-8b96-2235b760e26d" containerID="83ae84cff86b4ad4f18030ddbbf62c10e485e19b24e8fefc9bae8a150b77c735" exitCode=0 Jul 10 20:09:08.260186 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.260005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wshtg" event={"ID":"01fdc1af-61b3-4a78-8b96-2235b760e26d","Type":"ContainerDied","Data":"83ae84cff86b4ad4f18030ddbbf62c10e485e19b24e8fefc9bae8a150b77c735"} Jul 10 20:09:08.263717 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.263699 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:09:08.264102 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.264084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerStarted","Data":"e93ff9a54302bc83ec43d92839575337eeb12948e4e0e19417516e9d02cdb3c3"} Jul 10 20:09:08.264385 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.264363 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l676f" Jul 10 20:09:08.264575 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.264562 2565 scope.go:117] "RemoveContainer" containerID="4c4bf9e6f2306c7f79dd945f4f9c56bff4e0f979856fd2f385e3b148f6296b24" Jul 10 20:09:08.264761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:08.264743 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l676f" Jul 10 20:09:09.268152 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.268057 2565 generic.go:356] "Generic (PLEG): container finished" podID="01fdc1af-61b3-4a78-8b96-2235b760e26d" containerID="04cc060a5a3cbf9e228fb72b8e7939ea52c6a8e79f55575322088eaf7a498214" exitCode=0 Jul 10 20:09:09.268152 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.268092 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wshtg" event={"ID":"01fdc1af-61b3-4a78-8b96-2235b760e26d","Type":"ContainerDied","Data":"04cc060a5a3cbf9e228fb72b8e7939ea52c6a8e79f55575322088eaf7a498214"} Jul 10 20:09:09.272245 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.272224 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:09:09.272660 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.272639 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" event={"ID":"88851b04-9c40-4f05-a3f2-7d62c45e32db","Type":"ContainerStarted","Data":"ecac07d7a08996ee768805e0798deb5bb774b5e6549b5974164825ed9adfc7d4"} Jul 10 20:09:09.272815 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.272793 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:09:09.272938 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.272884 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:09:09.273089 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.273072 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:09:09.289738 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.289693 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:09:09.289897 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.289838 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:09:09.313790 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:09.313732 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" podStartSLOduration=9.828587219 podStartE2EDuration="29.313714371s" totalImagesPullingTime="19.485127152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:09.313367743 +0000 UTC m=+29.786337069" watchObservedRunningTime="2026-07-10 20:09:09.313714371 +0000 UTC m=+29.786683723" Jul 10 20:09:10.036155 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:10.035965 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:10.036352 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:10.036032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:10.036352 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:10.036226 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:10.036352 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:10.036322 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:10.036352 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:10.036060 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:10.036530 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:10.036425 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:10.277597 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:10.277512 2565 generic.go:356] "Generic (PLEG): container finished" podID="01fdc1af-61b3-4a78-8b96-2235b760e26d" containerID="3a9facf726edac49bb9c5d4a5f24bdc09f037a07ab9565af802b9bee94afdad1" exitCode=0 Jul 10 20:09:10.277982 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:10.277592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wshtg" event={"ID":"01fdc1af-61b3-4a78-8b96-2235b760e26d","Type":"ContainerDied","Data":"3a9facf726edac49bb9c5d4a5f24bdc09f037a07ab9565af802b9bee94afdad1"} Jul 10 20:09:12.035620 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:12.035579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:12.036091 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:12.035579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:12.036091 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:12.035579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:12.036550 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:12.036516 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:12.036687 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:12.036604 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:12.036745 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:12.036705 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:13.637451 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:13.637403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:13.637964 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:13.637546 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:09:13.637964 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:13.637613 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs podName:3ac597d8-dc79-46ad-954e-bc34432d454a nodeName:}" failed. No retries permitted until 2026-07-10 20:09:45.637596259 +0000 UTC m=+66.110565567 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs") pod "network-metrics-daemon-xfmzc" (UID: "3ac597d8-dc79-46ad-954e-bc34432d454a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 20:09:13.738736 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:13.738692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:13.738926 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:13.738881 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 20:09:13.738982 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:13.738956 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret podName:5e7a5331-206f-47f7-b6ee-237be41026bd nodeName:}" failed. No retries permitted until 2026-07-10 20:09:45.738939662 +0000 UTC m=+66.211908969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret") pod "global-pull-secret-syncer-m7gpw" (UID: "5e7a5331-206f-47f7-b6ee-237be41026bd") : object "kube-system"/"original-pull-secret" not registered Jul 10 20:09:13.839317 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:13.839282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:13.839495 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:13.839457 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 20:09:13.839495 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:13.839480 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 20:09:13.839495 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:13.839492 2565 projected.go:196] Error preparing data for projected volume kube-api-access-twkll for pod openshift-network-diagnostics/network-check-target-d62wz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:09:13.839638 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:13.839551 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll podName:4e7ce493-89e1-4d2d-ae2f-c57f227d2e26 nodeName:}" failed. No retries permitted until 2026-07-10 20:09:45.839533802 +0000 UTC m=+66.312503126 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-twkll" (UniqueName: "kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll") pod "network-check-target-d62wz" (UID: "4e7ce493-89e1-4d2d-ae2f-c57f227d2e26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 20:09:14.035090 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:14.035002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:14.035090 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:14.035037 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:14.035313 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:14.035002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:14.035313 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:14.035142 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:14.035313 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:14.035210 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:14.035456 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:14.035301 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:15.290843 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:15.290803 2565 generic.go:356] "Generic (PLEG): container finished" podID="01fdc1af-61b3-4a78-8b96-2235b760e26d" containerID="3f545fd74c83b3ce3b688d229d8b4c053ce56cc85641f194e95890244955120c" exitCode=0 Jul 10 20:09:15.291219 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:15.290891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wshtg" event={"ID":"01fdc1af-61b3-4a78-8b96-2235b760e26d","Type":"ContainerDied","Data":"3f545fd74c83b3ce3b688d229d8b4c053ce56cc85641f194e95890244955120c"} Jul 10 20:09:16.035581 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:16.035543 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:16.035581 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:16.035583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:16.035832 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:16.035545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:16.035832 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:16.035665 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:16.035832 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:16.035734 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:16.035832 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:16.035786 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:16.296136 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:16.296043 2565 generic.go:356] "Generic (PLEG): container finished" podID="01fdc1af-61b3-4a78-8b96-2235b760e26d" containerID="56f246f3e8b19c27c782aece133c9752d8f80ef6dee4fe5ccda3a954e2bf14e8" exitCode=0 Jul 10 20:09:16.296136 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:16.296102 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wshtg" event={"ID":"01fdc1af-61b3-4a78-8b96-2235b760e26d","Type":"ContainerDied","Data":"56f246f3e8b19c27c782aece133c9752d8f80ef6dee4fe5ccda3a954e2bf14e8"} Jul 10 20:09:17.301555 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:17.301518 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wshtg" event={"ID":"01fdc1af-61b3-4a78-8b96-2235b760e26d","Type":"ContainerStarted","Data":"49d010a20df1e2189bab68931045a39daed2620bf67c5b3624c97e5371405ea0"} Jul 10 20:09:17.325460 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:17.325406 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wshtg" podStartSLOduration=7.802870355 podStartE2EDuration="37.325390631s" totalImagesPullingTime="29.522520276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:17.32368018 +0000 UTC m=+37.796649507" watchObservedRunningTime="2026-07-10 20:09:17.325390631 +0000 UTC m=+37.798359957" Jul 10 20:09:18.035034 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:18.035002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:18.035223 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:18.035013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:18.035223 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:18.035118 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:18.035223 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:18.035013 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:18.036228 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:18.035602 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:18.036228 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:18.035766 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:20.032950 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:20.032917 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d62wz"] Jul 10 20:09:20.033497 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:20.033038 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:20.033497 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:20.033141 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:20.036322 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:20.036293 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:20.036489 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:20.036406 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:20.036489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:20.036459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:20.036594 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:20.036574 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:20.038991 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:20.038968 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m7gpw"] Jul 10 20:09:20.039093 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:20.039017 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xfmzc"] Jul 10 20:09:20.309552 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:20.309357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:20.309757 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:20.309651 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:20.312490 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:20.309816 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:20.312490 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:20.311188 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:22.035086 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:22.035044 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:22.035561 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:22.035044 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:22.035561 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:22.035182 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:22.035561 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:22.035044 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:22.035561 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:22.035228 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:22.035561 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:22.035305 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:24.034965 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.034927 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:24.034965 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.034947 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:24.035439 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.034935 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:24.035439 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:24.035049 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xfmzc" podUID="3ac597d8-dc79-46ad-954e-bc34432d454a" Jul 10 20:09:24.035439 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:24.035120 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m7gpw" podUID="5e7a5331-206f-47f7-b6ee-237be41026bd" Jul 10 20:09:24.035439 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:24.035239 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d62wz" podUID="4e7ce493-89e1-4d2d-ae2f-c57f227d2e26" Jul 10 20:09:24.354436 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.354348 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-108.ec2.internal" event="NodeReady" Jul 10 20:09:24.354587 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.354511 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 20:09:24.395838 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.395805 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rvf6x"] Jul 10 20:09:24.426403 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.426369 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bjfb5"] Jul 10 20:09:24.426536 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.426488 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvf6x" Jul 10 20:09:24.429184 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.429159 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n22ht\"" Jul 10 20:09:24.429322 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.429228 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 20:09:24.429376 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.429317 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 20:09:24.429376 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.429350 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 20:09:24.440468 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.440437 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvf6x"] Jul 10 20:09:24.440650 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.440479 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bjfb5"] Jul 10 20:09:24.440650 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.440513 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kvgvk"] Jul 10 20:09:24.440650 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.440558 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.443104 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.443039 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-djccr\"" Jul 10 20:09:24.443104 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.443076 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 20:09:24.443104 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.443083 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 20:09:24.443104 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.443089 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 20:09:24.443412 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.443399 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 20:09:24.462600 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.462566 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kvgvk"] Jul 10 20:09:24.462758 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.462687 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.465010 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.464984 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 20:09:24.465164 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.465122 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 20:09:24.465237 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.465171 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fsg7c\"" Jul 10 20:09:24.510431 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.510393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c782ed4-67a4-4d80-bcff-23e322671811-cert\") pod \"ingress-canary-rvf6x\" (UID: \"9c782ed4-67a4-4d80-bcff-23e322671811\") " pod="openshift-ingress-canary/ingress-canary-rvf6x" Jul 10 20:09:24.510431 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.510438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvzb\" (UniqueName: \"kubernetes.io/projected/9c782ed4-67a4-4d80-bcff-23e322671811-kube-api-access-hwvzb\") pod \"ingress-canary-rvf6x\" (UID: \"9c782ed4-67a4-4d80-bcff-23e322671811\") " pod="openshift-ingress-canary/ingress-canary-rvf6x" Jul 10 20:09:24.611799 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.611720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/411019d5-e164-41f1-91d8-ffd907a4319c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.611799 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.611753 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8ll\" (UniqueName: \"kubernetes.io/projected/411019d5-e164-41f1-91d8-ffd907a4319c-kube-api-access-jg8ll\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.611993 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.611840 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80f7b86b-a451-477c-940a-0ab9fbcf8f62-metrics-tls\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.611993 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.611890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/411019d5-e164-41f1-91d8-ffd907a4319c-crio-socket\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.611993 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.611918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8k8f\" (UniqueName: \"kubernetes.io/projected/80f7b86b-a451-477c-940a-0ab9fbcf8f62-kube-api-access-r8k8f\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.611993 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.611944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c782ed4-67a4-4d80-bcff-23e322671811-cert\") pod \"ingress-canary-rvf6x\" (UID: \"9c782ed4-67a4-4d80-bcff-23e322671811\") " pod="openshift-ingress-canary/ingress-canary-rvf6x" Jul 10 20:09:24.611993 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.611958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/411019d5-e164-41f1-91d8-ffd907a4319c-data-volume\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.612145 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.612085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80f7b86b-a451-477c-940a-0ab9fbcf8f62-config-volume\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.612145 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.612113 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/411019d5-e164-41f1-91d8-ffd907a4319c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.612145 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.612139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f7b86b-a451-477c-940a-0ab9fbcf8f62-tmp-dir\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.612282 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.612167 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvzb\" (UniqueName: \"kubernetes.io/projected/9c782ed4-67a4-4d80-bcff-23e322671811-kube-api-access-hwvzb\") pod \"ingress-canary-rvf6x\" (UID: \"9c782ed4-67a4-4d80-bcff-23e322671811\") " pod="openshift-ingress-canary/ingress-canary-rvf6x" Jul 10 20:09:24.616573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.616547 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c782ed4-67a4-4d80-bcff-23e322671811-cert\") pod \"ingress-canary-rvf6x\" (UID: \"9c782ed4-67a4-4d80-bcff-23e322671811\") " pod="openshift-ingress-canary/ingress-canary-rvf6x" Jul 10 20:09:24.620053 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.620028 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvzb\" (UniqueName: \"kubernetes.io/projected/9c782ed4-67a4-4d80-bcff-23e322671811-kube-api-access-hwvzb\") pod \"ingress-canary-rvf6x\" (UID: \"9c782ed4-67a4-4d80-bcff-23e322671811\") " pod="openshift-ingress-canary/ingress-canary-rvf6x" Jul 10 20:09:24.713109 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713068 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80f7b86b-a451-477c-940a-0ab9fbcf8f62-metrics-tls\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.713109 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/411019d5-e164-41f1-91d8-ffd907a4319c-crio-socket\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.713343 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8k8f\" (UniqueName: \"kubernetes.io/projected/80f7b86b-a451-477c-940a-0ab9fbcf8f62-kube-api-access-r8k8f\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.713343 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713158 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/411019d5-e164-41f1-91d8-ffd907a4319c-data-volume\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.713343 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80f7b86b-a451-477c-940a-0ab9fbcf8f62-config-volume\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.713343 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/411019d5-e164-41f1-91d8-ffd907a4319c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.713343 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/411019d5-e164-41f1-91d8-ffd907a4319c-crio-socket\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.713607 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f7b86b-a451-477c-940a-0ab9fbcf8f62-tmp-dir\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.713607 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/411019d5-e164-41f1-91d8-ffd907a4319c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.713607 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8ll\" (UniqueName: \"kubernetes.io/projected/411019d5-e164-41f1-91d8-ffd907a4319c-kube-api-access-jg8ll\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.713607 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/411019d5-e164-41f1-91d8-ffd907a4319c-data-volume\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.713792 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.713636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f7b86b-a451-477c-940a-0ab9fbcf8f62-tmp-dir\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.714030 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.714014 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/411019d5-e164-41f1-91d8-ffd907a4319c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.714180 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.714081 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/80f7b86b-a451-477c-940a-0ab9fbcf8f62-config-volume\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.715632 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.715615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/80f7b86b-a451-477c-940a-0ab9fbcf8f62-metrics-tls\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.715666 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.715644 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/411019d5-e164-41f1-91d8-ffd907a4319c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.721048 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.721026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8ll\" (UniqueName: \"kubernetes.io/projected/411019d5-e164-41f1-91d8-ffd907a4319c-kube-api-access-jg8ll\") pod \"insights-runtime-extractor-bjfb5\" (UID: \"411019d5-e164-41f1-91d8-ffd907a4319c\") " pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.731527 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.731496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8k8f\" (UniqueName: \"kubernetes.io/projected/80f7b86b-a451-477c-940a-0ab9fbcf8f62-kube-api-access-r8k8f\") pod \"dns-default-kvgvk\" (UID: \"80f7b86b-a451-477c-940a-0ab9fbcf8f62\") " pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.736331 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.736309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvf6x" Jul 10 20:09:24.749616 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.749582 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bjfb5" Jul 10 20:09:24.771488 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.771458 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:24.808177 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.807154 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-564cd47984-dtn72"] Jul 10 20:09:24.816520 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.816444 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:24.819538 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.819512 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 20:09:24.819693 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.819668 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 20:09:24.819970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.819951 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 20:09:24.820095 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.820070 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 20:09:24.820309 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.820292 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 20:09:24.820525 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.819710 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-66xc9\"" Jul 10 20:09:24.820585 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.820521 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 20:09:24.821254 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.821100 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 20:09:24.822259 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.822202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564cd47984-dtn72"] Jul 10 20:09:24.916523 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.916482 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr9rc\" (UniqueName: \"kubernetes.io/projected/276df272-ed2c-499c-abd9-102e1ef63893-kube-api-access-fr9rc\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:24.916666 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.916596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-oauth-serving-cert\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:24.916709 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.916659 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-service-ca\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:24.916739 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.916721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-serving-cert\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:24.916771 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.916745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-oauth-config\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:24.916802 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.916777 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-console-config\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:24.979285 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.979248 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvf6x"] Jul 10 20:09:24.983423 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:24.983384 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c782ed4_67a4_4d80_bcff_23e322671811.slice/crio-ada6c9faf889854fd8247d4b6c62f199ebf016c18bd7613f324f525781a5d8bf WatchSource:0}: Error finding container ada6c9faf889854fd8247d4b6c62f199ebf016c18bd7613f324f525781a5d8bf: Status 404 returned error can't find the container with id ada6c9faf889854fd8247d4b6c62f199ebf016c18bd7613f324f525781a5d8bf Jul 10 20:09:24.986223 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.986197 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bjfb5"] Jul 10 20:09:24.989497 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:24.989471 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411019d5_e164_41f1_91d8_ffd907a4319c.slice/crio-659d24842741394bff1e6ce78db0d3d233938f69eb5b8f552670a8b4dbee8e86 WatchSource:0}: Error finding container 659d24842741394bff1e6ce78db0d3d233938f69eb5b8f552670a8b4dbee8e86: Status 404 returned error can't find the container with id 659d24842741394bff1e6ce78db0d3d233938f69eb5b8f552670a8b4dbee8e86 Jul 10 20:09:24.991349 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:24.991323 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kvgvk"] Jul 10 20:09:24.995824 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:24.995796 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f7b86b_a451_477c_940a_0ab9fbcf8f62.slice/crio-a052b5af93a42524605505a3f02d1bfe8019f1e48d788faeeed04fe8e0927b02 WatchSource:0}: Error finding container a052b5af93a42524605505a3f02d1bfe8019f1e48d788faeeed04fe8e0927b02: Status 404 returned error can't find the container with id a052b5af93a42524605505a3f02d1bfe8019f1e48d788faeeed04fe8e0927b02 Jul 10 20:09:25.018133 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.018101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr9rc\" (UniqueName: \"kubernetes.io/projected/276df272-ed2c-499c-abd9-102e1ef63893-kube-api-access-fr9rc\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.018312 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.018174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-oauth-serving-cert\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.018312 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.018208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-service-ca\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.018312 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.018228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-serving-cert\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.018312 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.018250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-oauth-config\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.018312 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.018279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-console-config\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.019154 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.019127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-service-ca\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.019278 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.019255 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-console-config\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.019317 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.019261 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-oauth-serving-cert\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.021749 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.021719 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-oauth-config\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.021799 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.021760 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-serving-cert\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.025943 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.025923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr9rc\" (UniqueName: \"kubernetes.io/projected/276df272-ed2c-499c-abd9-102e1ef63893-kube-api-access-fr9rc\") pod \"console-564cd47984-dtn72\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.129894 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.129780 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:25.269733 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.269545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564cd47984-dtn72"] Jul 10 20:09:25.272254 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:25.272227 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276df272_ed2c_499c_abd9_102e1ef63893.slice/crio-50493f7d6ec57b320a53e202de390fe19a7515c7323241df85a5a52fedfe35cc WatchSource:0}: Error finding container 50493f7d6ec57b320a53e202de390fe19a7515c7323241df85a5a52fedfe35cc: Status 404 returned error can't find the container with id 50493f7d6ec57b320a53e202de390fe19a7515c7323241df85a5a52fedfe35cc Jul 10 20:09:25.321577 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.321538 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564cd47984-dtn72" event={"ID":"276df272-ed2c-499c-abd9-102e1ef63893","Type":"ContainerStarted","Data":"50493f7d6ec57b320a53e202de390fe19a7515c7323241df85a5a52fedfe35cc"} Jul 10 20:09:25.322689 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.322658 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kvgvk" event={"ID":"80f7b86b-a451-477c-940a-0ab9fbcf8f62","Type":"ContainerStarted","Data":"a052b5af93a42524605505a3f02d1bfe8019f1e48d788faeeed04fe8e0927b02"} Jul 10 20:09:25.323761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.323735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvf6x" event={"ID":"9c782ed4-67a4-4d80-bcff-23e322671811","Type":"ContainerStarted","Data":"ada6c9faf889854fd8247d4b6c62f199ebf016c18bd7613f324f525781a5d8bf"} Jul 10 20:09:25.324759 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:25.324739 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjfb5" event={"ID":"411019d5-e164-41f1-91d8-ffd907a4319c","Type":"ContainerStarted","Data":"659d24842741394bff1e6ce78db0d3d233938f69eb5b8f552670a8b4dbee8e86"} Jul 10 20:09:26.036028 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:26.035999 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:26.036216 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:26.036006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:26.036478 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:26.036457 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:26.040445 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:26.039676 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 20:09:26.040445 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:26.039747 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 20:09:26.040445 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:26.039953 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6l2jv\"" Jul 10 20:09:26.040445 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:26.040037 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2llrd\"" Jul 10 20:09:26.040445 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:26.040232 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 20:09:26.040445 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:26.040250 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 20:09:28.398538 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.398499 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d4b869c5f-ft2cc"] Jul 10 20:09:28.445255 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.445201 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d4b869c5f-ft2cc"] Jul 10 20:09:28.445448 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.445347 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.453246 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.453038 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 20:09:28.548726 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.548679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-config\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.548932 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.548753 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-oauth-serving-cert\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.548932 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.548915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-serving-cert\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.549022 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.548942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt6ww\" (UniqueName: \"kubernetes.io/projected/0e14b74e-3b90-4158-8036-5883e6a7eb93-kube-api-access-lt6ww\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.549022 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.548972 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-trusted-ca-bundle\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.549094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.549057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-oauth-config\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.549094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.549086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-service-ca\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.649775 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.649675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-serving-cert\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.649775 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.649754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt6ww\" (UniqueName: \"kubernetes.io/projected/0e14b74e-3b90-4158-8036-5883e6a7eb93-kube-api-access-lt6ww\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.650044 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.649787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-trusted-ca-bundle\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.650044 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.649822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-oauth-config\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.650044 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.649846 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-service-ca\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.650044 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.649891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-config\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.650044 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.649916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-oauth-serving-cert\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.650711 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.650683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-oauth-serving-cert\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.650855 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.650828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-service-ca\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.651074 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.650840 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-config\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.651148 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.651126 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-trusted-ca-bundle\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.653093 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.653063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-oauth-config\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.653093 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.653076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-serving-cert\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.658497 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.658463 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt6ww\" (UniqueName: \"kubernetes.io/projected/0e14b74e-3b90-4158-8036-5883e6a7eb93-kube-api-access-lt6ww\") pod \"console-d4b869c5f-ft2cc\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:28.756743 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:28.756702 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:32.768125 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:32.767759 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d4b869c5f-ft2cc"] Jul 10 20:09:32.785918 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:32.784960 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e14b74e_3b90_4158_8036_5883e6a7eb93.slice/crio-b3625e325cbee607ba76c82b68f22df4cc17d4989fb97b171d351048a32901ac WatchSource:0}: Error finding container b3625e325cbee607ba76c82b68f22df4cc17d4989fb97b171d351048a32901ac: Status 404 returned error can't find the container with id b3625e325cbee607ba76c82b68f22df4cc17d4989fb97b171d351048a32901ac Jul 10 20:09:32.817522 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:32.816527 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69"] Jul 10 20:09:32.819318 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:32.818598 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" Jul 10 20:09:32.824838 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:32.822754 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pdvcx\"" Jul 10 20:09:32.824838 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:32.823049 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 20:09:32.829920 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:32.828660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69"] Jul 10 20:09:32.983982 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:32.983937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ef09f15-f370-46c9-ba22-d0ee4c0acd6a-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-2pm69\" (UID: \"7ef09f15-f370-46c9-ba22-d0ee4c0acd6a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" Jul 10 20:09:33.085348 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.085306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ef09f15-f370-46c9-ba22-d0ee4c0acd6a-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-2pm69\" (UID: \"7ef09f15-f370-46c9-ba22-d0ee4c0acd6a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" Jul 10 20:09:33.085555 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:33.085452 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 10 20:09:33.085555 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:33.085528 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef09f15-f370-46c9-ba22-d0ee4c0acd6a-tls-certificates podName:7ef09f15-f370-46c9-ba22-d0ee4c0acd6a nodeName:}" failed. No retries permitted until 2026-07-10 20:09:33.585509672 +0000 UTC m=+54.058478980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7ef09f15-f370-46c9-ba22-d0ee4c0acd6a-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-2pm69" (UID: "7ef09f15-f370-46c9-ba22-d0ee4c0acd6a") : secret "prometheus-operator-admission-webhook-tls" not found Jul 10 20:09:33.352507 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.352470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kvgvk" event={"ID":"80f7b86b-a451-477c-940a-0ab9fbcf8f62","Type":"ContainerStarted","Data":"da62a543740a578dcca5008673c63af186666e4cdc7ce54a5fa94a039193ce67"} Jul 10 20:09:33.352690 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.352534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kvgvk" event={"ID":"80f7b86b-a451-477c-940a-0ab9fbcf8f62","Type":"ContainerStarted","Data":"f9c6898713d98364128b15f906748e24b442c6ce5efd51a705f58981cf071292"} Jul 10 20:09:33.352690 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.352599 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:33.354082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.354054 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvf6x" event={"ID":"9c782ed4-67a4-4d80-bcff-23e322671811","Type":"ContainerStarted","Data":"e7b56f69e19a38a4deec6f8d6383fbdcd3d10b4bfd8620d666f31936e9852064"} Jul 10 20:09:33.355924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.355887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjfb5" event={"ID":"411019d5-e164-41f1-91d8-ffd907a4319c","Type":"ContainerStarted","Data":"1d962dae1948451f741be6b02abf427bf0515f674cfbc9773c3b81bba4b3ebd1"} Jul 10 20:09:33.358259 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.358233 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4b869c5f-ft2cc" event={"ID":"0e14b74e-3b90-4158-8036-5883e6a7eb93","Type":"ContainerStarted","Data":"a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680"} Jul 10 20:09:33.358396 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.358282 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4b869c5f-ft2cc" event={"ID":"0e14b74e-3b90-4158-8036-5883e6a7eb93","Type":"ContainerStarted","Data":"b3625e325cbee607ba76c82b68f22df4cc17d4989fb97b171d351048a32901ac"} Jul 10 20:09:33.359992 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.359966 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564cd47984-dtn72" event={"ID":"276df272-ed2c-499c-abd9-102e1ef63893","Type":"ContainerStarted","Data":"1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81"} Jul 10 20:09:33.370572 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.370525 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kvgvk" podStartSLOduration=2.112888388 podStartE2EDuration="9.370507964s" totalImagesPullingTime="7.257619576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:33.369279446 +0000 UTC m=+53.842248772" watchObservedRunningTime="2026-07-10 20:09:33.370507964 +0000 UTC m=+53.843477290" Jul 10 20:09:33.384781 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.384715 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rvf6x" podStartSLOduration=1.760186051 podStartE2EDuration="9.384700811s" totalImagesPullingTime="7.62451476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:33.383625052 +0000 UTC m=+53.856594378" watchObservedRunningTime="2026-07-10 20:09:33.384700811 +0000 UTC m=+53.857670140" Jul 10 20:09:33.402077 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.402014 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-564cd47984-dtn72" podStartSLOduration=2.066562847 podStartE2EDuration="9.401991745s" totalImagesPullingTime="7.335428898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:33.401685156 +0000 UTC m=+53.874654551" watchObservedRunningTime="2026-07-10 20:09:33.401991745 +0000 UTC m=+53.874961072" Jul 10 20:09:33.416956 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.416899 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d4b869c5f-ft2cc" podStartSLOduration=5.416880984 podStartE2EDuration="5.416880984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:33.416771534 +0000 UTC m=+53.889740861" watchObservedRunningTime="2026-07-10 20:09:33.416880984 +0000 UTC m=+53.889850307" Jul 10 20:09:33.591640 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.591460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ef09f15-f370-46c9-ba22-d0ee4c0acd6a-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-2pm69\" (UID: \"7ef09f15-f370-46c9-ba22-d0ee4c0acd6a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" Jul 10 20:09:33.595773 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.595746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ef09f15-f370-46c9-ba22-d0ee4c0acd6a-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-2pm69\" (UID: \"7ef09f15-f370-46c9-ba22-d0ee4c0acd6a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" Jul 10 20:09:33.733545 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.733500 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" Jul 10 20:09:33.864089 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:33.864066 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69"] Jul 10 20:09:33.867587 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:33.867554 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef09f15_f370_46c9_ba22_d0ee4c0acd6a.slice/crio-1665086f4cbdbd8c7b93b436a3ff32ea08aa530bedfc4a8528e27402ec481973 WatchSource:0}: Error finding container 1665086f4cbdbd8c7b93b436a3ff32ea08aa530bedfc4a8528e27402ec481973: Status 404 returned error can't find the container with id 1665086f4cbdbd8c7b93b436a3ff32ea08aa530bedfc4a8528e27402ec481973 Jul 10 20:09:34.366504 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:34.366470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjfb5" event={"ID":"411019d5-e164-41f1-91d8-ffd907a4319c","Type":"ContainerStarted","Data":"431103ecd7e4170f5ad2a3aa0969a61cca008ca7cb5b18a07bbeb73ab46916a5"} Jul 10 20:09:34.367715 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:34.367675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" event={"ID":"7ef09f15-f370-46c9-ba22-d0ee4c0acd6a","Type":"ContainerStarted","Data":"1665086f4cbdbd8c7b93b436a3ff32ea08aa530bedfc4a8528e27402ec481973"} Jul 10 20:09:35.130460 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:35.130185 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:35.130942 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:35.130513 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:35.132023 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:35.131994 2565 patch_prober.go:28] interesting pod/console-564cd47984-dtn72 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 10 20:09:35.132144 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:35.132048 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-564cd47984-dtn72" podUID="276df272-ed2c-499c-abd9-102e1ef63893" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 10 20:09:36.376539 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:36.376498 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" event={"ID":"7ef09f15-f370-46c9-ba22-d0ee4c0acd6a","Type":"ContainerStarted","Data":"cdf75bc3d1c7adc98561f189361574ade8ea4317be9e727736f1d0804c42d51a"} Jul 10 20:09:36.377003 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:36.376676 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" Jul 10 20:09:36.378989 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:36.378954 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjfb5" event={"ID":"411019d5-e164-41f1-91d8-ffd907a4319c","Type":"ContainerStarted","Data":"b1fc18301dc3a496b04f24b593ffbb8c978cb4dc04ada3e8d142deb1bd959e42"} Jul 10 20:09:36.381813 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:36.381791 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" Jul 10 20:09:36.390398 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:36.390346 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-2pm69" podStartSLOduration=2.062149721 podStartE2EDuration="4.390327402s" totalImagesPullingTime="2.328177681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:36.389613429 +0000 UTC m=+56.862582755" watchObservedRunningTime="2026-07-10 20:09:36.390327402 +0000 UTC m=+56.863296729" Jul 10 20:09:36.419921 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:36.419857 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bjfb5" podStartSLOduration=1.371584041 podStartE2EDuration="12.419835248s" totalImagesPullingTime="11.048251207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:09:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:36.419406171 +0000 UTC m=+56.892375497" watchObservedRunningTime="2026-07-10 20:09:36.419835248 +0000 UTC m=+56.892804556" Jul 10 20:09:38.757279 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:38.757243 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:38.757659 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:38.757307 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:09:38.758549 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:38.758525 2565 patch_prober.go:28] interesting pod/console-d4b869c5f-ft2cc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 10 20:09:38.758673 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:38.758569 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-d4b869c5f-ft2cc" podUID="0e14b74e-3b90-4158-8036-5883e6a7eb93" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 10 20:09:41.291966 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:41.291929 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v8bxg" Jul 10 20:09:42.235791 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.235760 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk"] Jul 10 20:09:42.238574 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.238555 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.241269 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.241243 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 20:09:42.241468 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.241343 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 20:09:42.242005 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.241976 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 20:09:42.242110 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.242047 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dmb85\"" Jul 10 20:09:42.242172 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.241989 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 20:09:42.242283 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.242265 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 20:09:42.251996 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.251970 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk"] Jul 10 20:09:42.255983 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.255958 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gtj9v"] Jul 10 20:09:42.257246 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.257222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95ac415d-4e61-42fa-8bb6-524cde061644-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.257361 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.257256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ac415d-4e61-42fa-8bb6-524cde061644-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.257361 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.257276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6bf\" (UniqueName: \"kubernetes.io/projected/95ac415d-4e61-42fa-8bb6-524cde061644-kube-api-access-vl6bf\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.257475 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.257372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ac415d-4e61-42fa-8bb6-524cde061644-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.258054 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.258032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.260192 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.260164 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 20:09:42.260192 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.260171 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 20:09:42.260350 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.260207 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 20:09:42.260350 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.260231 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-twgm6\"" Jul 10 20:09:42.358069 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-textfile\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.358069 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358092 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ac415d-4e61-42fa-8bb6-524cde061644-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95ac415d-4e61-42fa-8bb6-524cde061644-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8adc0775-e750-4ace-92a9-b640c2d19417-sys\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358287 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4n6l\" (UniqueName: \"kubernetes.io/projected/8adc0775-e750-4ace-92a9-b640c2d19417-kube-api-access-m4n6l\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8adc0775-e750-4ace-92a9-b640c2d19417-metrics-client-ca\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ac415d-4e61-42fa-8bb6-524cde061644-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6bf\" (UniqueName: \"kubernetes.io/projected/95ac415d-4e61-42fa-8bb6-524cde061644-kube-api-access-vl6bf\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:42.358437 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8adc0775-e750-4ace-92a9-b640c2d19417-root\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.358489 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:42.358495 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ac415d-4e61-42fa-8bb6-524cde061644-openshift-state-metrics-tls podName:95ac415d-4e61-42fa-8bb6-524cde061644 nodeName:}" failed. No retries permitted until 2026-07-10 20:09:42.858476707 +0000 UTC m=+63.331446025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/95ac415d-4e61-42fa-8bb6-524cde061644-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-f8khk" (UID: "95ac415d-4e61-42fa-8bb6-524cde061644") : secret "openshift-state-metrics-tls" not found Jul 10 20:09:42.358914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-tls\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.358914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.358546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-wtmp\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.359033 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.359013 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95ac415d-4e61-42fa-8bb6-524cde061644-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.361905 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.361887 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95ac415d-4e61-42fa-8bb6-524cde061644-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.366107 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.366078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6bf\" (UniqueName: \"kubernetes.io/projected/95ac415d-4e61-42fa-8bb6-524cde061644-kube-api-access-vl6bf\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.459082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8adc0775-e750-4ace-92a9-b640c2d19417-root\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-tls\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459320 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-wtmp\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459320 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-textfile\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459320 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459160 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8adc0775-e750-4ace-92a9-b640c2d19417-root\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459320 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459320 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459320 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8adc0775-e750-4ace-92a9-b640c2d19417-sys\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459589 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4n6l\" (UniqueName: \"kubernetes.io/projected/8adc0775-e750-4ace-92a9-b640c2d19417-kube-api-access-m4n6l\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459589 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8adc0775-e750-4ace-92a9-b640c2d19417-sys\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459589 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459308 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-wtmp\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459589 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8adc0775-e750-4ace-92a9-b640c2d19417-metrics-client-ca\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.459781 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.459677 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-textfile\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.460067 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.460046 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.460140 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.460046 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8adc0775-e750-4ace-92a9-b640c2d19417-metrics-client-ca\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.462202 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.462178 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.462315 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.462209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8adc0775-e750-4ace-92a9-b640c2d19417-node-exporter-tls\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.470110 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.470087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4n6l\" (UniqueName: \"kubernetes.io/projected/8adc0775-e750-4ace-92a9-b640c2d19417-kube-api-access-m4n6l\") pod \"node-exporter-gtj9v\" (UID: \"8adc0775-e750-4ace-92a9-b640c2d19417\") " pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.567300 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.567206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gtj9v" Jul 10 20:09:42.577009 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:42.576979 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adc0775_e750_4ace_92a9_b640c2d19417.slice/crio-ffcd62c58fe11bac56e15bd58dd4c96e35d89337b024565c4373eef71cfeebf8 WatchSource:0}: Error finding container ffcd62c58fe11bac56e15bd58dd4c96e35d89337b024565c4373eef71cfeebf8: Status 404 returned error can't find the container with id ffcd62c58fe11bac56e15bd58dd4c96e35d89337b024565c4373eef71cfeebf8 Jul 10 20:09:42.862608 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.862524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ac415d-4e61-42fa-8bb6-524cde061644-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:42.865024 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:42.865004 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ac415d-4e61-42fa-8bb6-524cde061644-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-f8khk\" (UID: \"95ac415d-4e61-42fa-8bb6-524cde061644\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:43.148247 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:43.148209 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" Jul 10 20:09:43.304516 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:43.304486 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk"] Jul 10 20:09:43.307804 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:43.307772 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ac415d_4e61_42fa_8bb6_524cde061644.slice/crio-4e9d0f2264ed2a74dfed4a44fdd6f18ba8c1000997d118b009f8219010cfea7b WatchSource:0}: Error finding container 4e9d0f2264ed2a74dfed4a44fdd6f18ba8c1000997d118b009f8219010cfea7b: Status 404 returned error can't find the container with id 4e9d0f2264ed2a74dfed4a44fdd6f18ba8c1000997d118b009f8219010cfea7b Jul 10 20:09:43.369925 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:43.369899 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kvgvk" Jul 10 20:09:43.407067 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:43.406972 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtj9v" event={"ID":"8adc0775-e750-4ace-92a9-b640c2d19417","Type":"ContainerStarted","Data":"ffcd62c58fe11bac56e15bd58dd4c96e35d89337b024565c4373eef71cfeebf8"} Jul 10 20:09:43.410449 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:43.410400 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" event={"ID":"95ac415d-4e61-42fa-8bb6-524cde061644","Type":"ContainerStarted","Data":"7f0e2c2545c6ad331b1339c70e2e3da9351e0a1bdfafd08ec066b5295b99d6eb"} Jul 10 20:09:43.410610 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:43.410475 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" event={"ID":"95ac415d-4e61-42fa-8bb6-524cde061644","Type":"ContainerStarted","Data":"4e9d0f2264ed2a74dfed4a44fdd6f18ba8c1000997d118b009f8219010cfea7b"} Jul 10 20:09:44.415989 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:44.415948 2565 generic.go:356] "Generic (PLEG): container finished" podID="8adc0775-e750-4ace-92a9-b640c2d19417" containerID="b0d350c319f0eb5685c7291ac5766fe85fe935b9f9802109ca02a923551e99da" exitCode=0 Jul 10 20:09:44.416475 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:44.416041 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtj9v" event={"ID":"8adc0775-e750-4ace-92a9-b640c2d19417","Type":"ContainerDied","Data":"b0d350c319f0eb5685c7291ac5766fe85fe935b9f9802109ca02a923551e99da"} Jul 10 20:09:44.418051 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:44.418023 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" event={"ID":"95ac415d-4e61-42fa-8bb6-524cde061644","Type":"ContainerStarted","Data":"21bb980aa30829fde493c8afdecf0b79bc0c04305608b1b7263c3c5940764c4d"} Jul 10 20:09:45.134818 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.134788 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:45.138850 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.138824 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:09:45.202915 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.202882 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69d5d74485-pj2qz"] Jul 10 20:09:45.216274 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.216245 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.217389 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.217359 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69d5d74485-pj2qz"] Jul 10 20:09:45.218698 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.218669 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-631mus7d7jspt\"" Jul 10 20:09:45.218849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.218762 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 20:09:45.218849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.218791 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 20:09:45.218849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.218809 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 20:09:45.219019 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.218766 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 20:09:45.219085 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.219037 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 20:09:45.219137 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.219121 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qhvs8\"" Jul 10 20:09:45.287750 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.287720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2146e51-23dd-4d97-9a39-e15669b7db97-metrics-client-ca\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.287855 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.287800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.287855 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.287831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-grpc-tls\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.288022 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.287903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-tls\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.288022 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.287928 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.288022 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.287950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.288022 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.287994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmq9b\" (UniqueName: \"kubernetes.io/projected/e2146e51-23dd-4d97-9a39-e15669b7db97-kube-api-access-dmq9b\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.288205 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.288084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.389057 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.388974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-tls\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.389057 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.389010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.389057 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.389036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.389301 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.389072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmq9b\" (UniqueName: \"kubernetes.io/projected/e2146e51-23dd-4d97-9a39-e15669b7db97-kube-api-access-dmq9b\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.389301 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.389102 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.389301 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.389135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2146e51-23dd-4d97-9a39-e15669b7db97-metrics-client-ca\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.389301 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.389204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.389301 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.389234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-grpc-tls\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.390180 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.390148 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2146e51-23dd-4d97-9a39-e15669b7db97-metrics-client-ca\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.392055 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.392005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.392182 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.392150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.392182 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.392160 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.392367 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.392347 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.392664 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.392642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-grpc-tls\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.394109 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.394088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e2146e51-23dd-4d97-9a39-e15669b7db97-secret-thanos-querier-tls\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.397474 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.397452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmq9b\" (UniqueName: \"kubernetes.io/projected/e2146e51-23dd-4d97-9a39-e15669b7db97-kube-api-access-dmq9b\") pod \"thanos-querier-69d5d74485-pj2qz\" (UID: \"e2146e51-23dd-4d97-9a39-e15669b7db97\") " pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.423485 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.423449 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtj9v" event={"ID":"8adc0775-e750-4ace-92a9-b640c2d19417","Type":"ContainerStarted","Data":"aa30d6bb87efffc201b550c25336c9fce5397ce86fef67893f38cfd95ed32e1b"} Jul 10 20:09:45.423921 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.423503 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtj9v" event={"ID":"8adc0775-e750-4ace-92a9-b640c2d19417","Type":"ContainerStarted","Data":"7d013c30241a926a9f7bd24eb5f190dfe79f164ced5e53e338dc1cb0014f6889"} Jul 10 20:09:45.425594 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.425561 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" event={"ID":"95ac415d-4e61-42fa-8bb6-524cde061644","Type":"ContainerStarted","Data":"638ad310db47caf8fac670f63bddbea84c47700126bed8133d35a1a6ed24566e"} Jul 10 20:09:45.439388 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.439268 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gtj9v" podStartSLOduration=2.26984467 podStartE2EDuration="3.43924921s" totalImagesPullingTime="1.094445587s" totalInitContainerRuntime="74.958953ms" isStatefulPod=true podCreationTimestamp="2026-07-10 20:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:45.439055064 +0000 UTC m=+65.912024390" watchObservedRunningTime="2026-07-10 20:09:45.43924921 +0000 UTC m=+65.912218537" Jul 10 20:09:45.456624 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.456570 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-f8khk" podStartSLOduration=1.632007265 podStartE2EDuration="3.456553382s" totalImagesPullingTime="1.824546117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:45.455892356 +0000 UTC m=+65.928861684" watchObservedRunningTime="2026-07-10 20:09:45.456553382 +0000 UTC m=+65.929522707" Jul 10 20:09:45.527983 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.527950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:45.683165 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.683077 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69d5d74485-pj2qz"] Jul 10 20:09:45.686881 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:45.686821 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2146e51_23dd_4d97_9a39_e15669b7db97.slice/crio-72f737a1299d4e5dc42c9b0b3ebd8dd89ea1ccd3b6e27a11a6d8afd789d5083c WatchSource:0}: Error finding container 72f737a1299d4e5dc42c9b0b3ebd8dd89ea1ccd3b6e27a11a6d8afd789d5083c: Status 404 returned error can't find the container with id 72f737a1299d4e5dc42c9b0b3ebd8dd89ea1ccd3b6e27a11a6d8afd789d5083c Jul 10 20:09:45.691554 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.691533 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:45.693676 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.693653 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 20:09:45.705167 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.705142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac597d8-dc79-46ad-954e-bc34432d454a-metrics-certs\") pod \"network-metrics-daemon-xfmzc\" (UID: \"3ac597d8-dc79-46ad-954e-bc34432d454a\") " pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:45.792520 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.792477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:45.794707 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.794686 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 20:09:45.805894 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.805843 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5e7a5331-206f-47f7-b6ee-237be41026bd-original-pull-secret\") pod \"global-pull-secret-syncer-m7gpw\" (UID: \"5e7a5331-206f-47f7-b6ee-237be41026bd\") " pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:45.852249 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.852203 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m7gpw" Jul 10 20:09:45.863631 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.863597 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2llrd\"" Jul 10 20:09:45.871561 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.871526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xfmzc" Jul 10 20:09:45.893702 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.893643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:45.896273 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.896246 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 20:09:45.906588 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.906548 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 20:09:45.917711 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:45.917654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twkll\" (UniqueName: \"kubernetes.io/projected/4e7ce493-89e1-4d2d-ae2f-c57f227d2e26-kube-api-access-twkll\") pod \"network-check-target-d62wz\" (UID: \"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26\") " pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:46.012280 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.012249 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m7gpw"] Jul 10 20:09:46.015377 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:46.015342 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7a5331_206f_47f7_b6ee_237be41026bd.slice/crio-b8285a420383f65720a39d982344e8b01138c87e8b199ec668508ce880ce266d WatchSource:0}: Error finding container b8285a420383f65720a39d982344e8b01138c87e8b199ec668508ce880ce266d: Status 404 returned error can't find the container with id b8285a420383f65720a39d982344e8b01138c87e8b199ec668508ce880ce266d Jul 10 20:09:46.031454 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.031422 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xfmzc"] Jul 10 20:09:46.035179 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:46.035150 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac597d8_dc79_46ad_954e_bc34432d454a.slice/crio-161042cf5a7275f6770f77e56925a18afa745f396fd70c87a53bf9169a7b5157 WatchSource:0}: Error finding container 161042cf5a7275f6770f77e56925a18afa745f396fd70c87a53bf9169a7b5157: Status 404 returned error can't find the container with id 161042cf5a7275f6770f77e56925a18afa745f396fd70c87a53bf9169a7b5157 Jul 10 20:09:46.173931 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.173849 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6l2jv\"" Jul 10 20:09:46.179817 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.179788 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:46.344192 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.344152 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d62wz"] Jul 10 20:09:46.348612 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:46.348577 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7ce493_89e1_4d2d_ae2f_c57f227d2e26.slice/crio-795257f3a50c580e13d624530e1c6575d34c9c24f46237edf45bbf6204e27232 WatchSource:0}: Error finding container 795257f3a50c580e13d624530e1c6575d34c9c24f46237edf45bbf6204e27232: Status 404 returned error can't find the container with id 795257f3a50c580e13d624530e1c6575d34c9c24f46237edf45bbf6204e27232 Jul 10 20:09:46.431783 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.431741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d62wz" event={"ID":"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26","Type":"ContainerStarted","Data":"795257f3a50c580e13d624530e1c6575d34c9c24f46237edf45bbf6204e27232"} Jul 10 20:09:46.434400 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.434273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfmzc" event={"ID":"3ac597d8-dc79-46ad-954e-bc34432d454a","Type":"ContainerStarted","Data":"161042cf5a7275f6770f77e56925a18afa745f396fd70c87a53bf9169a7b5157"} Jul 10 20:09:46.435752 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.435694 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m7gpw" event={"ID":"5e7a5331-206f-47f7-b6ee-237be41026bd","Type":"ContainerStarted","Data":"b8285a420383f65720a39d982344e8b01138c87e8b199ec668508ce880ce266d"} Jul 10 20:09:46.437067 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.437040 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" event={"ID":"e2146e51-23dd-4d97-9a39-e15669b7db97","Type":"ContainerStarted","Data":"72f737a1299d4e5dc42c9b0b3ebd8dd89ea1ccd3b6e27a11a6d8afd789d5083c"} Jul 10 20:09:46.524680 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.523559 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65f947fd9c-7qwpw"] Jul 10 20:09:46.525819 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.525775 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.528482 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.528278 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 20:09:46.528482 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.528353 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-twtwt\"" Jul 10 20:09:46.528482 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.528364 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 20:09:46.528482 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.528278 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 20:09:46.528761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.528572 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 20:09:46.528761 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.528572 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ecb127at2lql1\"" Jul 10 20:09:46.537509 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.537478 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65f947fd9c-7qwpw"] Jul 10 20:09:46.599289 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.599245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-audit-log\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.599453 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.599308 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-client-ca-bundle\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.599453 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.599345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvrt\" (UniqueName: \"kubernetes.io/projected/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-kube-api-access-2lvrt\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.599453 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.599378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-secret-metrics-server-client-certs\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.599453 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.599411 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.599453 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.599437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-metrics-server-audit-profiles\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.599733 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.599555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-secret-metrics-server-tls\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.700674 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.700632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-audit-log\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.700837 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.700692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-client-ca-bundle\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.700837 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.700730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvrt\" (UniqueName: \"kubernetes.io/projected/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-kube-api-access-2lvrt\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.700837 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.700762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-secret-metrics-server-client-certs\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.700837 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.700794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.700837 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.700820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-metrics-server-audit-profiles\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.701156 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.700901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-secret-metrics-server-tls\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.701156 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.701098 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-audit-log\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.702197 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.702171 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.702893 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.702846 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-metrics-server-audit-profiles\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.704784 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.704753 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-secret-metrics-server-client-certs\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.705628 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.705601 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-client-ca-bundle\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.706775 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.706750 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-secret-metrics-server-tls\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.709460 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.709435 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvrt\" (UniqueName: \"kubernetes.io/projected/6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10-kube-api-access-2lvrt\") pod \"metrics-server-65f947fd9c-7qwpw\" (UID: \"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10\") " pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:46.839502 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:46.839421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:09:47.004763 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.003529 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg"] Jul 10 20:09:47.005982 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.005956 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:47.010143 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.009955 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 20:09:47.010143 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.010046 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8q4t5\"" Jul 10 20:09:47.016547 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.016474 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg"] Jul 10 20:09:47.024686 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.024656 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d4b869c5f-ft2cc"] Jul 10 20:09:47.044630 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.044596 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74766f8cb5-lkvfw"] Jul 10 20:09:47.047017 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.046996 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.058683 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.058608 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74766f8cb5-lkvfw"] Jul 10 20:09:47.107968 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.107739 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-trusted-ca-bundle\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.107968 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.107793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-serving-cert\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.107968 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.107830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-oauth-config\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.107968 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.107879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-oauth-serving-cert\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.108280 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.107963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06e4a08d-aacf-4197-86a4-e2d654ec33ba-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xfhvg\" (UID: \"06e4a08d-aacf-4197-86a4-e2d654ec33ba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:47.108280 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.108009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwrs\" (UniqueName: \"kubernetes.io/projected/9af28d4a-b29b-40ef-8e94-d58417981093-kube-api-access-6vwrs\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.108280 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.108045 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlppp\" (UniqueName: \"kubernetes.io/projected/06e4a08d-aacf-4197-86a4-e2d654ec33ba-kube-api-access-dlppp\") pod \"monitoring-plugin-5f68df7f58-xfhvg\" (UID: \"06e4a08d-aacf-4197-86a4-e2d654ec33ba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:47.108280 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.108096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-console-config\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.108280 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.108141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-service-ca\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.211047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.209552 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-serving-cert\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.211047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.209613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-oauth-config\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.211047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.209642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-oauth-serving-cert\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.211047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.209680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06e4a08d-aacf-4197-86a4-e2d654ec33ba-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xfhvg\" (UID: \"06e4a08d-aacf-4197-86a4-e2d654ec33ba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:47.211047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.209711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwrs\" (UniqueName: \"kubernetes.io/projected/9af28d4a-b29b-40ef-8e94-d58417981093-kube-api-access-6vwrs\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.211047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.209748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlppp\" (UniqueName: \"kubernetes.io/projected/06e4a08d-aacf-4197-86a4-e2d654ec33ba-kube-api-access-dlppp\") pod \"monitoring-plugin-5f68df7f58-xfhvg\" (UID: \"06e4a08d-aacf-4197-86a4-e2d654ec33ba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:47.211047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.209790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-console-config\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.211047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.209825 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-service-ca\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.211047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.209896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-trusted-ca-bundle\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.214983 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.212294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-console-config\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.214983 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.213155 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-oauth-serving-cert\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.214983 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:47.213251 2565 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 10 20:09:47.214983 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:09:47.213309 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06e4a08d-aacf-4197-86a4-e2d654ec33ba-monitoring-plugin-cert podName:06e4a08d-aacf-4197-86a4-e2d654ec33ba nodeName:}" failed. No retries permitted until 2026-07-10 20:09:47.713289214 +0000 UTC m=+68.186258518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/06e4a08d-aacf-4197-86a4-e2d654ec33ba-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-xfhvg" (UID: "06e4a08d-aacf-4197-86a4-e2d654ec33ba") : secret "monitoring-plugin-cert" not found Jul 10 20:09:47.214983 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.214105 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-service-ca\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.214983 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.214906 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-trusted-ca-bundle\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.215836 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.215788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-oauth-config\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.216668 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.216476 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-serving-cert\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.227404 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.227340 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlppp\" (UniqueName: \"kubernetes.io/projected/06e4a08d-aacf-4197-86a4-e2d654ec33ba-kube-api-access-dlppp\") pod \"monitoring-plugin-5f68df7f58-xfhvg\" (UID: \"06e4a08d-aacf-4197-86a4-e2d654ec33ba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:47.229927 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.229845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwrs\" (UniqueName: \"kubernetes.io/projected/9af28d4a-b29b-40ef-8e94-d58417981093-kube-api-access-6vwrs\") pod \"console-74766f8cb5-lkvfw\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.287330 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.287232 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65f947fd9c-7qwpw"] Jul 10 20:09:47.361383 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.361061 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:47.446786 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.446177 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfmzc" event={"ID":"3ac597d8-dc79-46ad-954e-bc34432d454a","Type":"ContainerStarted","Data":"228259540bccb8061de9e4f4bd847d3baf6b57db17b04c1e91fee818f7f53ddb"} Jul 10 20:09:47.446786 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.446235 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xfmzc" event={"ID":"3ac597d8-dc79-46ad-954e-bc34432d454a","Type":"ContainerStarted","Data":"6cb36a6e4086467cddfee2acbcf9953ee29b6e7caf373c5e73b9e08e510eb905"} Jul 10 20:09:47.449017 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.448981 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" event={"ID":"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10","Type":"ContainerStarted","Data":"f0a6968f9ae575d0ecbd888997f7fcf573095d2296a633351bda7d0b0705a73c"} Jul 10 20:09:47.462591 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.462522 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xfmzc" podStartSLOduration=66.377417231 podStartE2EDuration="1m7.462497026s" totalImagesPullingTime="1.085079795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:47.462490316 +0000 UTC m=+67.935459743" watchObservedRunningTime="2026-07-10 20:09:47.462497026 +0000 UTC m=+67.935466352" Jul 10 20:09:47.715666 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.715630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06e4a08d-aacf-4197-86a4-e2d654ec33ba-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xfhvg\" (UID: \"06e4a08d-aacf-4197-86a4-e2d654ec33ba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:47.718507 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.718477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/06e4a08d-aacf-4197-86a4-e2d654ec33ba-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xfhvg\" (UID: \"06e4a08d-aacf-4197-86a4-e2d654ec33ba\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:47.921328 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:47.921293 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:48.186037 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.186002 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg"] Jul 10 20:09:48.207379 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.207341 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74766f8cb5-lkvfw"] Jul 10 20:09:48.213951 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:48.213924 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af28d4a_b29b_40ef_8e94_d58417981093.slice/crio-9ca4ec27d8f956124c57203f30e473834e7cf7f3604c80f93d48c0bf71db89a7 WatchSource:0}: Error finding container 9ca4ec27d8f956124c57203f30e473834e7cf7f3604c80f93d48c0bf71db89a7: Status 404 returned error can't find the container with id 9ca4ec27d8f956124c57203f30e473834e7cf7f3604c80f93d48c0bf71db89a7 Jul 10 20:09:48.419759 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.417414 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 20:09:48.422112 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.421588 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.426151 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.426171 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.426421 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.426465 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.426593 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.426770 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.426895 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dsonp0lgvnih7\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.427000 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.427042 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.427129 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 20:09:48.427627 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.427431 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 20:09:48.437418 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.433664 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 20:09:48.437418 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.434193 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 20:09:48.437418 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.434267 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kb8gw\"" Jul 10 20:09:48.437418 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.436548 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 20:09:48.437418 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.437437 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 20:09:48.440601 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.438031 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 20:09:48.458032 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.457998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" event={"ID":"e2146e51-23dd-4d97-9a39-e15669b7db97","Type":"ContainerStarted","Data":"63264d0d35e5c3a82a29cd116d77ac73356eade6775266ff8726fa0ea3340015"} Jul 10 20:09:48.458497 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.458086 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" event={"ID":"e2146e51-23dd-4d97-9a39-e15669b7db97","Type":"ContainerStarted","Data":"c94c0275ceb4fcde42ac1d57338b5f9ef0b6296045dbb382f4c8560b9a822538"} Jul 10 20:09:48.458497 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.458129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" event={"ID":"e2146e51-23dd-4d97-9a39-e15669b7db97","Type":"ContainerStarted","Data":"70d73d60f02b972567b57d289e030ece4d7c15864ca690c85dc3c31d6d069194"} Jul 10 20:09:48.459792 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.459762 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" event={"ID":"06e4a08d-aacf-4197-86a4-e2d654ec33ba","Type":"ContainerStarted","Data":"6b24d4755fe421d4a7472da1035628692320832e4d1a26b9b2a7c423d8df2ebb"} Jul 10 20:09:48.462268 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.461884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74766f8cb5-lkvfw" event={"ID":"9af28d4a-b29b-40ef-8e94-d58417981093","Type":"ContainerStarted","Data":"99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de"} Jul 10 20:09:48.462268 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.461938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74766f8cb5-lkvfw" event={"ID":"9af28d4a-b29b-40ef-8e94-d58417981093","Type":"ContainerStarted","Data":"9ca4ec27d8f956124c57203f30e473834e7cf7f3604c80f93d48c0bf71db89a7"} Jul 10 20:09:48.487620 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.487283 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74766f8cb5-lkvfw" podStartSLOduration=1.4872654060000001 podStartE2EDuration="1.487265406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:48.484933264 +0000 UTC m=+68.957902590" watchObservedRunningTime="2026-07-10 20:09:48.487265406 +0000 UTC m=+68.960234710" Jul 10 20:09:48.526326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.525822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.525914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.525950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.525993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5km\" (UniqueName: \"kubernetes.io/projected/c5bff782-4bbe-45a3-8739-1645dc1d0617-kube-api-access-tc5km\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5bff782-4bbe-45a3-8739-1645dc1d0617-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526326 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526144 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5bff782-4bbe-45a3-8739-1645dc1d0617-config-out\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526898 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526898 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-web-config\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526898 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526898 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526635 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526898 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526898 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526751 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.526898 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.527191 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-config\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.527191 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.526990 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.527191 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.527018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.527191 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.527054 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628480 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5km\" (UniqueName: \"kubernetes.io/projected/c5bff782-4bbe-45a3-8739-1645dc1d0617-kube-api-access-tc5km\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5bff782-4bbe-45a3-8739-1645dc1d0617-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5bff782-4bbe-45a3-8739-1645dc1d0617-config-out\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-web-config\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628846 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.629336 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628904 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-config\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.630461 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.630461 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.630461 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.628998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.632971 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.630701 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.632971 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.630701 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.632971 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.631053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.632971 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.631260 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.632971 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.631716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.635465 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.635426 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5bff782-4bbe-45a3-8739-1645dc1d0617-config-out\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.639304 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.638368 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5bff782-4bbe-45a3-8739-1645dc1d0617-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.639304 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.638667 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.639304 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.638803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.639304 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.639258 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.639804 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.639313 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.640611 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.640552 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.641094 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.640688 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c5bff782-4bbe-45a3-8739-1645dc1d0617-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.641725 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.641685 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.641725 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.641685 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.641951 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.641704 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-web-config\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.642478 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.642454 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-config\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.643938 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.643820 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.647512 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.647047 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5bff782-4bbe-45a3-8739-1645dc1d0617-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.650444 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.650390 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5km\" (UniqueName: \"kubernetes.io/projected/c5bff782-4bbe-45a3-8739-1645dc1d0617-kube-api-access-tc5km\") pod \"prometheus-k8s-0\" (UID: \"c5bff782-4bbe-45a3-8739-1645dc1d0617\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.741264 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.741227 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:09:48.947443 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:48.945880 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 20:09:48.950709 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:48.950212 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bff782_4bbe_45a3_8739_1645dc1d0617.slice/crio-3233b8947f210753c46f1a68a13b9cc8adc58e272abc5135aac12285bc8a4ef6 WatchSource:0}: Error finding container 3233b8947f210753c46f1a68a13b9cc8adc58e272abc5135aac12285bc8a4ef6: Status 404 returned error can't find the container with id 3233b8947f210753c46f1a68a13b9cc8adc58e272abc5135aac12285bc8a4ef6 Jul 10 20:09:49.468564 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:49.468523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5bff782-4bbe-45a3-8739-1645dc1d0617","Type":"ContainerStarted","Data":"3233b8947f210753c46f1a68a13b9cc8adc58e272abc5135aac12285bc8a4ef6"} Jul 10 20:09:52.082282 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.082237 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74766f8cb5-lkvfw"] Jul 10 20:09:52.119800 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.119765 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b9b75498-pg8n8"] Jul 10 20:09:52.124396 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.124365 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.133989 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.133958 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9b75498-pg8n8"] Jul 10 20:09:52.164417 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.164371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-console-config\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.164610 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.164430 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-oauth-config\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.164610 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.164468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-serving-cert\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.164610 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.164482 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-service-ca\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.164610 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.164534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cxm9\" (UniqueName: \"kubernetes.io/projected/98d2a3c0-343e-4e89-864e-9202708c63d7-kube-api-access-9cxm9\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.164610 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.164579 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-oauth-serving-cert\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.164855 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.164629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-trusted-ca-bundle\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.265751 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.265705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-serving-cert\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.265751 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.265754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-service-ca\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.266041 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.265795 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cxm9\" (UniqueName: \"kubernetes.io/projected/98d2a3c0-343e-4e89-864e-9202708c63d7-kube-api-access-9cxm9\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.266041 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.265817 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-oauth-serving-cert\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.266041 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.265907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-trusted-ca-bundle\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.266041 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.265979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-console-config\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.266041 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.266000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-oauth-config\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.266615 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.266585 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-service-ca\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.266796 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.266775 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-trusted-ca-bundle\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.266937 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.266915 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-oauth-serving-cert\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.266985 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.266928 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-console-config\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.268886 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.268756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-oauth-config\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.269055 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.269037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-serving-cert\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.274257 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.274231 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cxm9\" (UniqueName: \"kubernetes.io/projected/98d2a3c0-343e-4e89-864e-9202708c63d7-kube-api-access-9cxm9\") pod \"console-5b9b75498-pg8n8\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.438313 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.438273 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:09:52.661467 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:52.661431 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9b75498-pg8n8"] Jul 10 20:09:52.812102 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:09:52.812072 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d2a3c0_343e_4e89_864e_9202708c63d7.slice/crio-bb9e8688ab303fa2a324c7b9157d5a2da2d628b91eae3d0206eb9b39d508ad42 WatchSource:0}: Error finding container bb9e8688ab303fa2a324c7b9157d5a2da2d628b91eae3d0206eb9b39d508ad42: Status 404 returned error can't find the container with id bb9e8688ab303fa2a324c7b9157d5a2da2d628b91eae3d0206eb9b39d508ad42 Jul 10 20:09:53.488023 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.487988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d62wz" event={"ID":"4e7ce493-89e1-4d2d-ae2f-c57f227d2e26","Type":"ContainerStarted","Data":"45c1b690bd5f05b511dd4b23e84cd2b730e621acaa28a317c940ee06dc1f6648"} Jul 10 20:09:53.488471 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.488079 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:09:53.489504 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.489481 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m7gpw" event={"ID":"5e7a5331-206f-47f7-b6ee-237be41026bd","Type":"ContainerStarted","Data":"057eb2c6660a61f6ca98c794c5bf57178b0ec3bc47c89a7713f5a567df81c68d"} Jul 10 20:09:53.492539 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.492518 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" event={"ID":"e2146e51-23dd-4d97-9a39-e15669b7db97","Type":"ContainerStarted","Data":"5b5f18581a56821aed6d4479d9c04ef0938e18aefbdab1aea3ca2d63b8b8b1e4"} Jul 10 20:09:53.492654 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.492567 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" event={"ID":"e2146e51-23dd-4d97-9a39-e15669b7db97","Type":"ContainerStarted","Data":"97c153e49d55c03d7a50b199e18070f0b3fd2c2075e5a792abbb644c3729bd6e"} Jul 10 20:09:53.492654 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.492604 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" event={"ID":"e2146e51-23dd-4d97-9a39-e15669b7db97","Type":"ContainerStarted","Data":"1486406b8f5ff3d3b6b7ef075367b8a848b14e6c1747ef4f7a2cee320fe0955f"} Jul 10 20:09:53.492654 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.492642 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:53.494198 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.494168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" event={"ID":"6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10","Type":"ContainerStarted","Data":"13f367598ea96b46843e835d873cc933e79d39ea5c22a6903070d9e68547512d"} Jul 10 20:09:53.500065 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.500033 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" event={"ID":"06e4a08d-aacf-4197-86a4-e2d654ec33ba","Type":"ContainerStarted","Data":"e5b2d5cf54ff0c9bda21f8c42c7e23c128fff8316723f3294eb65d7ecbe4cb24"} Jul 10 20:09:53.500200 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.500117 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:53.502576 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.502531 2565 generic.go:356] "Generic (PLEG): container finished" podID="c5bff782-4bbe-45a3-8739-1645dc1d0617" containerID="fb3a8bebdff5d7b629578711b4c6c4bf4c2eb688a874ab2cfdf46865709d8e15" exitCode=0 Jul 10 20:09:53.503474 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.503446 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5bff782-4bbe-45a3-8739-1645dc1d0617","Type":"ContainerDied","Data":"fb3a8bebdff5d7b629578711b4c6c4bf4c2eb688a874ab2cfdf46865709d8e15"} Jul 10 20:09:53.504250 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.504232 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" Jul 10 20:09:53.505210 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.505151 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d62wz" podStartSLOduration=67.046234249 podStartE2EDuration="1m13.505133563s" totalImagesPullingTime="6.458899314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:53.502089222 +0000 UTC m=+73.975058551" watchObservedRunningTime="2026-07-10 20:09:53.505133563 +0000 UTC m=+73.978102889" Jul 10 20:09:53.505435 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.505413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9b75498-pg8n8" event={"ID":"98d2a3c0-343e-4e89-864e-9202708c63d7","Type":"ContainerStarted","Data":"5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280"} Jul 10 20:09:53.505515 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.505457 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9b75498-pg8n8" event={"ID":"98d2a3c0-343e-4e89-864e-9202708c63d7","Type":"ContainerStarted","Data":"bb9e8688ab303fa2a324c7b9157d5a2da2d628b91eae3d0206eb9b39d508ad42"} Jul 10 20:09:53.506364 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.506347 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" Jul 10 20:09:53.521120 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.521051 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m7gpw" podStartSLOduration=66.704197515 podStartE2EDuration="1m13.521034866s" totalImagesPullingTime="6.816837351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:53.520104584 +0000 UTC m=+73.993073910" watchObservedRunningTime="2026-07-10 20:09:53.521034866 +0000 UTC m=+73.994004169" Jul 10 20:09:53.541657 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.541168 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69d5d74485-pj2qz" podStartSLOduration=2.004664199 podStartE2EDuration="8.541147783s" totalImagesPullingTime="6.536483584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:53.540183345 +0000 UTC m=+74.013152672" watchObservedRunningTime="2026-07-10 20:09:53.541147783 +0000 UTC m=+74.014117113" Jul 10 20:09:53.556526 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.556472 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xfhvg" podStartSLOduration=3.2327061009999998 podStartE2EDuration="7.556456136s" totalImagesPullingTime="4.323750035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:53.554676651 +0000 UTC m=+74.027645979" watchObservedRunningTime="2026-07-10 20:09:53.556456136 +0000 UTC m=+74.029425440" Jul 10 20:09:53.595367 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.595310 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" podStartSLOduration=2.373262789 podStartE2EDuration="7.595292103s" totalImagesPullingTime="5.222029314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:53.573673633 +0000 UTC m=+74.046642986" watchObservedRunningTime="2026-07-10 20:09:53.595292103 +0000 UTC m=+74.068261432" Jul 10 20:09:53.653623 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:53.653550 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b9b75498-pg8n8" podStartSLOduration=1.653531909 podStartE2EDuration="1.653531909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:53.652076386 +0000 UTC m=+74.125045715" watchObservedRunningTime="2026-07-10 20:09:53.653531909 +0000 UTC m=+74.126501236" Jul 10 20:09:57.362066 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:57.361975 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:09:57.527740 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:57.527703 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5bff782-4bbe-45a3-8739-1645dc1d0617","Type":"ContainerStarted","Data":"febf20692a740da4e24b7cdb20d82071374aff37d2b19f87740af0c8ead6e2b1"} Jul 10 20:09:57.527900 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:57.527770 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5bff782-4bbe-45a3-8739-1645dc1d0617","Type":"ContainerStarted","Data":"cb708eda51ccd3e55ccc76d35841166096f0f34d22cbb8cf0a0d95598f180a7a"} Jul 10 20:09:57.527900 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:57.527799 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5bff782-4bbe-45a3-8739-1645dc1d0617","Type":"ContainerStarted","Data":"a7628e2c6b875a2877b2e034d68083a4c2916731a6e68f833279e6d123ba4c43"} Jul 10 20:09:57.527900 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:57.527826 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5bff782-4bbe-45a3-8739-1645dc1d0617","Type":"ContainerStarted","Data":"0707d3f1cc7e155bb8a91fccf564b06430c0d18b41cfb34b10254395bdd3c06b"} Jul 10 20:09:57.527900 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:57.527849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5bff782-4bbe-45a3-8739-1645dc1d0617","Type":"ContainerStarted","Data":"02391643965a9ae00e59643470cb54823507b2a91bd26d1e8661bbc85382aac1"} Jul 10 20:09:57.528059 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:57.527917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5bff782-4bbe-45a3-8739-1645dc1d0617","Type":"ContainerStarted","Data":"6223094605b405d03f967134a108c2ae25c8193976c33963f3fdfec290eca11b"} Jul 10 20:09:57.554130 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:57.554066 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.301893402 podStartE2EDuration="9.554042101s" totalImagesPullingTime="7.215988376s" totalInitContainerRuntime="36.160323ms" isStatefulPod=false podCreationTimestamp="2026-07-10 20:09:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:09:57.552235936 +0000 UTC m=+78.025205263" watchObservedRunningTime="2026-07-10 20:09:57.554042101 +0000 UTC m=+78.027011431" Jul 10 20:09:58.742150 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:09:58.742107 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:10:02.439367 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:02.439228 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:10:02.439367 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:02.439313 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:10:02.444416 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:02.444391 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:10:02.551407 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:02.551377 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:10:02.594641 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:02.594614 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564cd47984-dtn72"] Jul 10 20:10:06.840406 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:06.840360 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:10:06.840780 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:06.840431 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:10:12.055519 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.055443 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d4b869c5f-ft2cc" podUID="0e14b74e-3b90-4158-8036-5883e6a7eb93" containerName="console" containerID="cri-o://a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680" gracePeriod=15 Jul 10 20:10:12.307850 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.307775 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:10:12.452388 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.452347 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-serving-cert\") pod \"0e14b74e-3b90-4158-8036-5883e6a7eb93\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " Jul 10 20:10:12.452584 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.452407 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-oauth-config\") pod \"0e14b74e-3b90-4158-8036-5883e6a7eb93\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " Jul 10 20:10:12.452584 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.452443 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-config\") pod \"0e14b74e-3b90-4158-8036-5883e6a7eb93\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " Jul 10 20:10:12.452584 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.452536 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-oauth-serving-cert\") pod \"0e14b74e-3b90-4158-8036-5883e6a7eb93\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " Jul 10 20:10:12.452584 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.452574 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-trusted-ca-bundle\") pod \"0e14b74e-3b90-4158-8036-5883e6a7eb93\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " Jul 10 20:10:12.452806 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.452616 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt6ww\" (UniqueName: \"kubernetes.io/projected/0e14b74e-3b90-4158-8036-5883e6a7eb93-kube-api-access-lt6ww\") pod \"0e14b74e-3b90-4158-8036-5883e6a7eb93\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " Jul 10 20:10:12.452806 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.452648 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-service-ca\") pod \"0e14b74e-3b90-4158-8036-5883e6a7eb93\" (UID: \"0e14b74e-3b90-4158-8036-5883e6a7eb93\") " Jul 10 20:10:12.452999 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.452969 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-config" (OuterVolumeSpecName: "console-config") pod "0e14b74e-3b90-4158-8036-5883e6a7eb93" (UID: "0e14b74e-3b90-4158-8036-5883e6a7eb93"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:12.453088 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.452992 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0e14b74e-3b90-4158-8036-5883e6a7eb93" (UID: "0e14b74e-3b90-4158-8036-5883e6a7eb93"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:12.453137 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.453079 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0e14b74e-3b90-4158-8036-5883e6a7eb93" (UID: "0e14b74e-3b90-4158-8036-5883e6a7eb93"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:12.453299 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.453278 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-service-ca" (OuterVolumeSpecName: "service-ca") pod "0e14b74e-3b90-4158-8036-5883e6a7eb93" (UID: "0e14b74e-3b90-4158-8036-5883e6a7eb93"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:12.453394 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.453289 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:12.453394 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.453316 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-oauth-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:12.453394 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.453329 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-trusted-ca-bundle\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:12.455004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.454980 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0e14b74e-3b90-4158-8036-5883e6a7eb93" (UID: "0e14b74e-3b90-4158-8036-5883e6a7eb93"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:10:12.455398 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.455378 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e14b74e-3b90-4158-8036-5883e6a7eb93-kube-api-access-lt6ww" (OuterVolumeSpecName: "kube-api-access-lt6ww") pod "0e14b74e-3b90-4158-8036-5883e6a7eb93" (UID: "0e14b74e-3b90-4158-8036-5883e6a7eb93"). InnerVolumeSpecName "kube-api-access-lt6ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:10:12.455474 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.455389 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0e14b74e-3b90-4158-8036-5883e6a7eb93" (UID: "0e14b74e-3b90-4158-8036-5883e6a7eb93"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:10:12.554368 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.554335 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:12.554368 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.554368 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e14b74e-3b90-4158-8036-5883e6a7eb93-console-oauth-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:12.554565 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.554383 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt6ww\" (UniqueName: \"kubernetes.io/projected/0e14b74e-3b90-4158-8036-5883e6a7eb93-kube-api-access-lt6ww\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:12.554565 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.554397 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e14b74e-3b90-4158-8036-5883e6a7eb93-service-ca\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:12.589964 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.589837 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e14b74e-3b90-4158-8036-5883e6a7eb93" containerID="a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680" exitCode=0 Jul 10 20:10:12.589964 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.589932 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d4b869c5f-ft2cc" Jul 10 20:10:12.589964 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.589926 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4b869c5f-ft2cc" event={"ID":"0e14b74e-3b90-4158-8036-5883e6a7eb93","Type":"ContainerDied","Data":"a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680"} Jul 10 20:10:12.590170 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.590017 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d4b869c5f-ft2cc" event={"ID":"0e14b74e-3b90-4158-8036-5883e6a7eb93","Type":"ContainerDied","Data":"b3625e325cbee607ba76c82b68f22df4cc17d4989fb97b171d351048a32901ac"} Jul 10 20:10:12.590170 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.590055 2565 scope.go:117] "RemoveContainer" containerID="a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680" Jul 10 20:10:12.600787 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.600769 2565 scope.go:117] "RemoveContainer" containerID="a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680" Jul 10 20:10:12.601239 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:10:12.601214 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680\": container with ID starting with a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680 not found: ID does not exist" containerID="a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680" Jul 10 20:10:12.601332 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.601247 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680"} err="failed to get container status \"a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680\": rpc error: code = NotFound desc = could not find container \"a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680\": container with ID starting with a6c47a870c82bf7045c75caa9ae408570d2c8c0cd424f7a17d408ce3de3cb680 not found: ID does not exist" Jul 10 20:10:12.615932 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.615903 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d4b869c5f-ft2cc"] Jul 10 20:10:12.619317 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:12.619290 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d4b869c5f-ft2cc"] Jul 10 20:10:14.040053 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:14.040023 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e14b74e-3b90-4158-8036-5883e6a7eb93" path="/var/lib/kubelet/pods/0e14b74e-3b90-4158-8036-5883e6a7eb93/volumes" Jul 10 20:10:17.106149 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.106091 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74766f8cb5-lkvfw" podUID="9af28d4a-b29b-40ef-8e94-d58417981093" containerName="console" containerID="cri-o://99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de" gracePeriod=15 Jul 10 20:10:17.130970 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.130946 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rvf6x_9c782ed4-67a4-4d80-bcff-23e322671811/serve-healthcheck-canary/0.log" Jul 10 20:10:17.353693 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.353661 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:10:17.499334 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.499295 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-oauth-config\") pod \"9af28d4a-b29b-40ef-8e94-d58417981093\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " Jul 10 20:10:17.499334 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.499344 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-service-ca\") pod \"9af28d4a-b29b-40ef-8e94-d58417981093\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " Jul 10 20:10:17.499577 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.499364 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vwrs\" (UniqueName: \"kubernetes.io/projected/9af28d4a-b29b-40ef-8e94-d58417981093-kube-api-access-6vwrs\") pod \"9af28d4a-b29b-40ef-8e94-d58417981093\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " Jul 10 20:10:17.499577 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.499392 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-serving-cert\") pod \"9af28d4a-b29b-40ef-8e94-d58417981093\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " Jul 10 20:10:17.499577 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.499419 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-oauth-serving-cert\") pod \"9af28d4a-b29b-40ef-8e94-d58417981093\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " Jul 10 20:10:17.499577 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.499440 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-trusted-ca-bundle\") pod \"9af28d4a-b29b-40ef-8e94-d58417981093\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " Jul 10 20:10:17.499577 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.499454 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-console-config\") pod \"9af28d4a-b29b-40ef-8e94-d58417981093\" (UID: \"9af28d4a-b29b-40ef-8e94-d58417981093\") " Jul 10 20:10:17.499929 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.499898 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-service-ca" (OuterVolumeSpecName: "service-ca") pod "9af28d4a-b29b-40ef-8e94-d58417981093" (UID: "9af28d4a-b29b-40ef-8e94-d58417981093"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:17.500025 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.499975 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9af28d4a-b29b-40ef-8e94-d58417981093" (UID: "9af28d4a-b29b-40ef-8e94-d58417981093"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:17.500025 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.500001 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-console-config" (OuterVolumeSpecName: "console-config") pod "9af28d4a-b29b-40ef-8e94-d58417981093" (UID: "9af28d4a-b29b-40ef-8e94-d58417981093"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:17.500109 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.500038 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9af28d4a-b29b-40ef-8e94-d58417981093" (UID: "9af28d4a-b29b-40ef-8e94-d58417981093"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:17.501936 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.501848 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9af28d4a-b29b-40ef-8e94-d58417981093" (UID: "9af28d4a-b29b-40ef-8e94-d58417981093"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:10:17.502042 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.501993 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af28d4a-b29b-40ef-8e94-d58417981093-kube-api-access-6vwrs" (OuterVolumeSpecName: "kube-api-access-6vwrs") pod "9af28d4a-b29b-40ef-8e94-d58417981093" (UID: "9af28d4a-b29b-40ef-8e94-d58417981093"). InnerVolumeSpecName "kube-api-access-6vwrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:10:17.502200 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.502179 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9af28d4a-b29b-40ef-8e94-d58417981093" (UID: "9af28d4a-b29b-40ef-8e94-d58417981093"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:10:17.600587 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.600528 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-oauth-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:17.600587 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.600587 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-service-ca\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:17.600784 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.600603 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vwrs\" (UniqueName: \"kubernetes.io/projected/9af28d4a-b29b-40ef-8e94-d58417981093-kube-api-access-6vwrs\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:17.600784 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.600617 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9af28d4a-b29b-40ef-8e94-d58417981093-console-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:17.600784 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.600631 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-oauth-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:17.600784 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.600647 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-trusted-ca-bundle\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:17.600784 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.600662 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9af28d4a-b29b-40ef-8e94-d58417981093-console-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:17.617461 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.617425 2565 generic.go:356] "Generic (PLEG): container finished" podID="9af28d4a-b29b-40ef-8e94-d58417981093" containerID="99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de" exitCode=0 Jul 10 20:10:17.617633 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.617498 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74766f8cb5-lkvfw" event={"ID":"9af28d4a-b29b-40ef-8e94-d58417981093","Type":"ContainerDied","Data":"99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de"} Jul 10 20:10:17.617633 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.617504 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74766f8cb5-lkvfw" Jul 10 20:10:17.617633 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.617560 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74766f8cb5-lkvfw" event={"ID":"9af28d4a-b29b-40ef-8e94-d58417981093","Type":"ContainerDied","Data":"9ca4ec27d8f956124c57203f30e473834e7cf7f3604c80f93d48c0bf71db89a7"} Jul 10 20:10:17.617633 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.617579 2565 scope.go:117] "RemoveContainer" containerID="99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de" Jul 10 20:10:17.626127 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.626099 2565 scope.go:117] "RemoveContainer" containerID="99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de" Jul 10 20:10:17.626442 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:10:17.626412 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de\": container with ID starting with 99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de not found: ID does not exist" containerID="99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de" Jul 10 20:10:17.626518 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.626452 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de"} err="failed to get container status \"99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de\": rpc error: code = NotFound desc = could not find container \"99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de\": container with ID starting with 99dd113cc763bec25c1ba4c22cf2133fde4e5dadd53ab08ee9365b9ad70a51de not found: ID does not exist" Jul 10 20:10:17.643513 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.643480 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74766f8cb5-lkvfw"] Jul 10 20:10:17.649344 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:17.649319 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74766f8cb5-lkvfw"] Jul 10 20:10:18.039603 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:18.039570 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9af28d4a-b29b-40ef-8e94-d58417981093" path="/var/lib/kubelet/pods/9af28d4a-b29b-40ef-8e94-d58417981093/volumes" Jul 10 20:10:24.513605 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:24.513566 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d62wz" Jul 10 20:10:26.845382 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:26.845352 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:10:26.849443 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:26.849412 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65f947fd9c-7qwpw" Jul 10 20:10:27.623074 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.623012 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-564cd47984-dtn72" podUID="276df272-ed2c-499c-abd9-102e1ef63893" containerName="console" containerID="cri-o://1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81" gracePeriod=15 Jul 10 20:10:27.898293 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.898257 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:10:27.988523 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.988489 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr9rc\" (UniqueName: \"kubernetes.io/projected/276df272-ed2c-499c-abd9-102e1ef63893-kube-api-access-fr9rc\") pod \"276df272-ed2c-499c-abd9-102e1ef63893\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " Jul 10 20:10:27.988702 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.988540 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-console-config\") pod \"276df272-ed2c-499c-abd9-102e1ef63893\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " Jul 10 20:10:27.988702 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.988572 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-oauth-config\") pod \"276df272-ed2c-499c-abd9-102e1ef63893\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " Jul 10 20:10:27.988702 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.988647 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-serving-cert\") pod \"276df272-ed2c-499c-abd9-102e1ef63893\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " Jul 10 20:10:27.988702 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.988677 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-oauth-serving-cert\") pod \"276df272-ed2c-499c-abd9-102e1ef63893\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " Jul 10 20:10:27.988937 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.988712 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-service-ca\") pod \"276df272-ed2c-499c-abd9-102e1ef63893\" (UID: \"276df272-ed2c-499c-abd9-102e1ef63893\") " Jul 10 20:10:27.988997 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.988940 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-console-config" (OuterVolumeSpecName: "console-config") pod "276df272-ed2c-499c-abd9-102e1ef63893" (UID: "276df272-ed2c-499c-abd9-102e1ef63893"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:27.989284 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.989254 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-service-ca" (OuterVolumeSpecName: "service-ca") pod "276df272-ed2c-499c-abd9-102e1ef63893" (UID: "276df272-ed2c-499c-abd9-102e1ef63893"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:27.989573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.989465 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "276df272-ed2c-499c-abd9-102e1ef63893" (UID: "276df272-ed2c-499c-abd9-102e1ef63893"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:10:27.991260 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.991225 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "276df272-ed2c-499c-abd9-102e1ef63893" (UID: "276df272-ed2c-499c-abd9-102e1ef63893"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:10:27.991393 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.991279 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "276df272-ed2c-499c-abd9-102e1ef63893" (UID: "276df272-ed2c-499c-abd9-102e1ef63893"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:10:27.999220 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:27.999183 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276df272-ed2c-499c-abd9-102e1ef63893-kube-api-access-fr9rc" (OuterVolumeSpecName: "kube-api-access-fr9rc") pod "276df272-ed2c-499c-abd9-102e1ef63893" (UID: "276df272-ed2c-499c-abd9-102e1ef63893"). InnerVolumeSpecName "kube-api-access-fr9rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:10:28.090100 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.090062 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:28.090267 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.090121 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-oauth-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:28.090267 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.090133 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-service-ca\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:28.090267 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.090143 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr9rc\" (UniqueName: \"kubernetes.io/projected/276df272-ed2c-499c-abd9-102e1ef63893-kube-api-access-fr9rc\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:28.090267 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.090154 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276df272-ed2c-499c-abd9-102e1ef63893-console-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:28.090267 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.090163 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276df272-ed2c-499c-abd9-102e1ef63893-console-oauth-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:10:28.664299 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.664247 2565 generic.go:356] "Generic (PLEG): container finished" podID="276df272-ed2c-499c-abd9-102e1ef63893" containerID="1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81" exitCode=0 Jul 10 20:10:28.664503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.664357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564cd47984-dtn72" event={"ID":"276df272-ed2c-499c-abd9-102e1ef63893","Type":"ContainerDied","Data":"1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81"} Jul 10 20:10:28.664503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.664454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564cd47984-dtn72" event={"ID":"276df272-ed2c-499c-abd9-102e1ef63893","Type":"ContainerDied","Data":"50493f7d6ec57b320a53e202de390fe19a7515c7323241df85a5a52fedfe35cc"} Jul 10 20:10:28.664503 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.664487 2565 scope.go:117] "RemoveContainer" containerID="1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81" Jul 10 20:10:28.664678 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.664667 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564cd47984-dtn72" Jul 10 20:10:28.674996 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.674855 2565 scope.go:117] "RemoveContainer" containerID="1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81" Jul 10 20:10:28.675266 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:10:28.675243 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81\": container with ID starting with 1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81 not found: ID does not exist" containerID="1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81" Jul 10 20:10:28.675308 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.675277 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81"} err="failed to get container status \"1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81\": rpc error: code = NotFound desc = could not find container \"1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81\": container with ID starting with 1bfe54445b7887b831a7fdfa7dcbbb5dfb66dcdde59390fd8067234df6eb1a81 not found: ID does not exist" Jul 10 20:10:28.680624 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.680595 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564cd47984-dtn72"] Jul 10 20:10:28.683803 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:28.683779 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-564cd47984-dtn72"] Jul 10 20:10:30.040646 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:30.040605 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276df272-ed2c-499c-abd9-102e1ef63893" path="/var/lib/kubelet/pods/276df272-ed2c-499c-abd9-102e1ef63893/volumes" Jul 10 20:10:48.741797 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:48.741762 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:10:48.762121 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:48.762085 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:10:49.764647 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:10:49.764614 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 20:11:04.987901 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.987845 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fbf9484b4-mrxjk"] Jul 10 20:11:04.988365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.988150 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276df272-ed2c-499c-abd9-102e1ef63893" containerName="console" Jul 10 20:11:04.988365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.988163 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="276df272-ed2c-499c-abd9-102e1ef63893" containerName="console" Jul 10 20:11:04.988365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.988174 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af28d4a-b29b-40ef-8e94-d58417981093" containerName="console" Jul 10 20:11:04.988365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.988179 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af28d4a-b29b-40ef-8e94-d58417981093" containerName="console" Jul 10 20:11:04.988365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.988187 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e14b74e-3b90-4158-8036-5883e6a7eb93" containerName="console" Jul 10 20:11:04.988365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.988193 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e14b74e-3b90-4158-8036-5883e6a7eb93" containerName="console" Jul 10 20:11:04.988365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.988252 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e14b74e-3b90-4158-8036-5883e6a7eb93" containerName="console" Jul 10 20:11:04.988365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.988261 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="276df272-ed2c-499c-abd9-102e1ef63893" containerName="console" Jul 10 20:11:04.988365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.988268 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af28d4a-b29b-40ef-8e94-d58417981093" containerName="console" Jul 10 20:11:04.990158 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:04.990140 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.003226 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.003192 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbf9484b4-mrxjk"] Jul 10 20:11:05.118243 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.118199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2pdb\" (UniqueName: \"kubernetes.io/projected/8c24bdf5-18e8-4186-8544-d6019620a1d9-kube-api-access-k2pdb\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.118243 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.118249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-service-ca\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.118463 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.118275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-serving-cert\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.118463 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.118316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-oauth-config\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.118463 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.118390 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-trusted-ca-bundle\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.118463 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.118446 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-config\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.118586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.118474 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-oauth-serving-cert\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.219116 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.219076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-oauth-config\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.219215 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.219132 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-trusted-ca-bundle\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.219215 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.219171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-config\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.219215 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.219197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-oauth-serving-cert\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.219312 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.219222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2pdb\" (UniqueName: \"kubernetes.io/projected/8c24bdf5-18e8-4186-8544-d6019620a1d9-kube-api-access-k2pdb\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.219312 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.219255 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-service-ca\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.219399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.219366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-serving-cert\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.220070 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.220038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-service-ca\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.220219 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.220088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-oauth-serving-cert\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.220297 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.220275 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-config\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.220352 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.220301 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-trusted-ca-bundle\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.221798 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.221777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-oauth-config\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.221932 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.221915 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-serving-cert\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.227052 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.227030 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2pdb\" (UniqueName: \"kubernetes.io/projected/8c24bdf5-18e8-4186-8544-d6019620a1d9-kube-api-access-k2pdb\") pod \"console-6fbf9484b4-mrxjk\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.300081 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.299986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:05.443169 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.443140 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbf9484b4-mrxjk"] Jul 10 20:11:05.445838 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:11:05.445806 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c24bdf5_18e8_4186_8544_d6019620a1d9.slice/crio-672a800a87e4334ab2472005c27c1603cf97c60599674d2aae94f3737dd2b0d3 WatchSource:0}: Error finding container 672a800a87e4334ab2472005c27c1603cf97c60599674d2aae94f3737dd2b0d3: Status 404 returned error can't find the container with id 672a800a87e4334ab2472005c27c1603cf97c60599674d2aae94f3737dd2b0d3 Jul 10 20:11:05.812527 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.812486 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbf9484b4-mrxjk" event={"ID":"8c24bdf5-18e8-4186-8544-d6019620a1d9","Type":"ContainerStarted","Data":"8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e"} Jul 10 20:11:05.812705 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.812547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbf9484b4-mrxjk" event={"ID":"8c24bdf5-18e8-4186-8544-d6019620a1d9","Type":"ContainerStarted","Data":"672a800a87e4334ab2472005c27c1603cf97c60599674d2aae94f3737dd2b0d3"} Jul 10 20:11:05.828852 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:05.828797 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fbf9484b4-mrxjk" podStartSLOduration=1.828778417 podStartE2EDuration="1.828778417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:11:05.827619114 +0000 UTC m=+146.300588444" watchObservedRunningTime="2026-07-10 20:11:05.828778417 +0000 UTC m=+146.301747743" Jul 10 20:11:15.300532 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:15.300481 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:15.301065 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:15.300556 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:15.305389 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:15.305360 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:15.851958 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:15.851926 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:11:15.899410 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:15.899374 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9b75498-pg8n8"] Jul 10 20:11:40.922787 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:40.922694 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b9b75498-pg8n8" podUID="98d2a3c0-343e-4e89-864e-9202708c63d7" containerName="console" containerID="cri-o://5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280" gracePeriod=15 Jul 10 20:11:41.171675 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.171651 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:11:41.346556 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.346452 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-trusted-ca-bundle\") pod \"98d2a3c0-343e-4e89-864e-9202708c63d7\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " Jul 10 20:11:41.346556 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.346508 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-oauth-config\") pod \"98d2a3c0-343e-4e89-864e-9202708c63d7\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " Jul 10 20:11:41.346556 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.346537 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-console-config\") pod \"98d2a3c0-343e-4e89-864e-9202708c63d7\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " Jul 10 20:11:41.346831 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.346706 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-service-ca\") pod \"98d2a3c0-343e-4e89-864e-9202708c63d7\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " Jul 10 20:11:41.346831 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.346787 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-oauth-serving-cert\") pod \"98d2a3c0-343e-4e89-864e-9202708c63d7\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " Jul 10 20:11:41.346974 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.346839 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-serving-cert\") pod \"98d2a3c0-343e-4e89-864e-9202708c63d7\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " Jul 10 20:11:41.346974 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.346899 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cxm9\" (UniqueName: \"kubernetes.io/projected/98d2a3c0-343e-4e89-864e-9202708c63d7-kube-api-access-9cxm9\") pod \"98d2a3c0-343e-4e89-864e-9202708c63d7\" (UID: \"98d2a3c0-343e-4e89-864e-9202708c63d7\") " Jul 10 20:11:41.347072 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.346936 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "98d2a3c0-343e-4e89-864e-9202708c63d7" (UID: "98d2a3c0-343e-4e89-864e-9202708c63d7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:11:41.347072 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.346976 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-console-config" (OuterVolumeSpecName: "console-config") pod "98d2a3c0-343e-4e89-864e-9202708c63d7" (UID: "98d2a3c0-343e-4e89-864e-9202708c63d7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:11:41.347161 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.347137 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-service-ca" (OuterVolumeSpecName: "service-ca") pod "98d2a3c0-343e-4e89-864e-9202708c63d7" (UID: "98d2a3c0-343e-4e89-864e-9202708c63d7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:11:41.347222 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.347149 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "98d2a3c0-343e-4e89-864e-9202708c63d7" (UID: "98d2a3c0-343e-4e89-864e-9202708c63d7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:11:41.347222 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.347171 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-trusted-ca-bundle\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:11:41.347222 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.347190 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-console-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:11:41.348896 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.348874 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "98d2a3c0-343e-4e89-864e-9202708c63d7" (UID: "98d2a3c0-343e-4e89-864e-9202708c63d7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:11:41.349104 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.349081 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d2a3c0-343e-4e89-864e-9202708c63d7-kube-api-access-9cxm9" (OuterVolumeSpecName: "kube-api-access-9cxm9") pod "98d2a3c0-343e-4e89-864e-9202708c63d7" (UID: "98d2a3c0-343e-4e89-864e-9202708c63d7"). InnerVolumeSpecName "kube-api-access-9cxm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:11:41.349176 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.349132 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "98d2a3c0-343e-4e89-864e-9202708c63d7" (UID: "98d2a3c0-343e-4e89-864e-9202708c63d7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:11:41.447655 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.447598 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:11:41.447655 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.447649 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cxm9\" (UniqueName: \"kubernetes.io/projected/98d2a3c0-343e-4e89-864e-9202708c63d7-kube-api-access-9cxm9\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:11:41.447655 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.447663 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d2a3c0-343e-4e89-864e-9202708c63d7-console-oauth-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:11:41.447927 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.447676 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-service-ca\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:11:41.447927 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.447689 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d2a3c0-343e-4e89-864e-9202708c63d7-oauth-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:11:41.944912 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.944853 2565 generic.go:356] "Generic (PLEG): container finished" podID="98d2a3c0-343e-4e89-864e-9202708c63d7" containerID="5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280" exitCode=0 Jul 10 20:11:41.945364 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.944922 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9b75498-pg8n8" Jul 10 20:11:41.945364 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.944931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9b75498-pg8n8" event={"ID":"98d2a3c0-343e-4e89-864e-9202708c63d7","Type":"ContainerDied","Data":"5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280"} Jul 10 20:11:41.945364 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.944984 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9b75498-pg8n8" event={"ID":"98d2a3c0-343e-4e89-864e-9202708c63d7","Type":"ContainerDied","Data":"bb9e8688ab303fa2a324c7b9157d5a2da2d628b91eae3d0206eb9b39d508ad42"} Jul 10 20:11:41.945364 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.945017 2565 scope.go:117] "RemoveContainer" containerID="5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280" Jul 10 20:11:41.954150 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.954133 2565 scope.go:117] "RemoveContainer" containerID="5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280" Jul 10 20:11:41.954436 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:11:41.954415 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280\": container with ID starting with 5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280 not found: ID does not exist" containerID="5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280" Jul 10 20:11:41.954513 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.954448 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280"} err="failed to get container status \"5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280\": rpc error: code = NotFound desc = could not find container \"5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280\": container with ID starting with 5e461625bb693b4a54a2eb7f7b5e93cc523589c2e59971adce8c83db7dcb8280 not found: ID does not exist" Jul 10 20:11:41.964681 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.964642 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9b75498-pg8n8"] Jul 10 20:11:41.969690 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:41.969664 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b9b75498-pg8n8"] Jul 10 20:11:42.039457 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:11:42.039426 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98d2a3c0-343e-4e89-864e-9202708c63d7" path="/var/lib/kubelet/pods/98d2a3c0-343e-4e89-864e-9202708c63d7/volumes" Jul 10 20:12:43.758490 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.758451 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh"] Jul 10 20:12:43.758935 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.758850 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d2a3c0-343e-4e89-864e-9202708c63d7" containerName="console" Jul 10 20:12:43.758935 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.758882 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d2a3c0-343e-4e89-864e-9202708c63d7" containerName="console" Jul 10 20:12:43.759019 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.758936 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d2a3c0-343e-4e89-864e-9202708c63d7" containerName="console" Jul 10 20:12:43.762135 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.762118 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" Jul 10 20:12:43.764125 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.764103 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 20:12:43.764978 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.764960 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-j24j5\"" Jul 10 20:12:43.765156 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.765144 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 20:12:43.770628 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.770601 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh"] Jul 10 20:12:43.869786 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.869738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71d97f2e-7b89-4f83-8db2-a5398bb9d355-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gnkgh\" (UID: \"71d97f2e-7b89-4f83-8db2-a5398bb9d355\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" Jul 10 20:12:43.870017 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.869849 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxmf\" (UniqueName: \"kubernetes.io/projected/71d97f2e-7b89-4f83-8db2-a5398bb9d355-kube-api-access-zcxmf\") pod \"openshift-lws-operator-fd8ccff4c-gnkgh\" (UID: \"71d97f2e-7b89-4f83-8db2-a5398bb9d355\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" Jul 10 20:12:43.971095 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.971045 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxmf\" (UniqueName: \"kubernetes.io/projected/71d97f2e-7b89-4f83-8db2-a5398bb9d355-kube-api-access-zcxmf\") pod \"openshift-lws-operator-fd8ccff4c-gnkgh\" (UID: \"71d97f2e-7b89-4f83-8db2-a5398bb9d355\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" Jul 10 20:12:43.971258 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.971134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71d97f2e-7b89-4f83-8db2-a5398bb9d355-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gnkgh\" (UID: \"71d97f2e-7b89-4f83-8db2-a5398bb9d355\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" Jul 10 20:12:43.971504 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.971482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71d97f2e-7b89-4f83-8db2-a5398bb9d355-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gnkgh\" (UID: \"71d97f2e-7b89-4f83-8db2-a5398bb9d355\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" Jul 10 20:12:43.978796 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:43.978770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxmf\" (UniqueName: \"kubernetes.io/projected/71d97f2e-7b89-4f83-8db2-a5398bb9d355-kube-api-access-zcxmf\") pod \"openshift-lws-operator-fd8ccff4c-gnkgh\" (UID: \"71d97f2e-7b89-4f83-8db2-a5398bb9d355\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" Jul 10 20:12:44.080560 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:44.080462 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" Jul 10 20:12:44.217007 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:44.216983 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh"] Jul 10 20:12:44.219708 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:12:44.219679 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71d97f2e_7b89_4f83_8db2_a5398bb9d355.slice/crio-3a011009269abc57042d2f104eceaed501cffcec16d285249e430fe037433f0a WatchSource:0}: Error finding container 3a011009269abc57042d2f104eceaed501cffcec16d285249e430fe037433f0a: Status 404 returned error can't find the container with id 3a011009269abc57042d2f104eceaed501cffcec16d285249e430fe037433f0a Jul 10 20:12:45.179311 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:45.179268 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" event={"ID":"71d97f2e-7b89-4f83-8db2-a5398bb9d355","Type":"ContainerStarted","Data":"3a011009269abc57042d2f104eceaed501cffcec16d285249e430fe037433f0a"} Jul 10 20:12:47.189459 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:47.189366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" event={"ID":"71d97f2e-7b89-4f83-8db2-a5398bb9d355","Type":"ContainerStarted","Data":"64c3fe989eb635208bcccffad56b06a4dc6633e3b40e3ce6bdbb98e4d2a6cf0f"} Jul 10 20:12:47.205603 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:12:47.205548 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gnkgh" podStartSLOduration=1.5405509450000001 podStartE2EDuration="4.205530465s" totalImagesPullingTime="2.66497952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:12:47.204696999 +0000 UTC m=+247.677666338" watchObservedRunningTime="2026-07-10 20:12:47.205530465 +0000 UTC m=+247.678499792" Jul 10 20:13:06.514895 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.514808 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827"] Jul 10 20:13:06.518217 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.518194 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.520555 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.520529 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qrfp4\"" Jul 10 20:13:06.520701 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.520570 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 20:13:06.520701 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.520603 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 20:13:06.520701 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.520616 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 20:13:06.521003 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.520987 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 20:13:06.531086 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.531061 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827"] Jul 10 20:13:06.678754 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.678717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-tmp\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.678754 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.678760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-manifests\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.679000 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.678819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-webhook-cert\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.679000 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.678888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.679000 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.678945 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2s5\" (UniqueName: \"kubernetes.io/projected/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-kube-api-access-ml2s5\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.780204 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.780118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-webhook-cert\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.780204 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.780167 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.780428 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.780297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2s5\" (UniqueName: \"kubernetes.io/projected/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-kube-api-access-ml2s5\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.780428 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.780383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-tmp\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.780428 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.780414 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-manifests\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.780770 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.780742 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-manifests\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.782728 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.782702 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-tmp\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.782844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.782759 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.782964 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.782946 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-webhook-cert\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.788088 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.788063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2s5\" (UniqueName: \"kubernetes.io/projected/f53333fc-9d3a-4bbc-aca6-5d2d2cb36495-kube-api-access-ml2s5\") pod \"opendatahub-operator-controller-manager-5767ffdff4-fv827\" (UID: \"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495\") " pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.828018 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.827976 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:06.973932 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:06.973752 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827"] Jul 10 20:13:06.976978 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:13:06.976944 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53333fc_9d3a_4bbc_aca6_5d2d2cb36495.slice/crio-084d2d66e1f581d89b945ef59d66ce61b18c43b1cb9a906be551f304a8b487ee WatchSource:0}: Error finding container 084d2d66e1f581d89b945ef59d66ce61b18c43b1cb9a906be551f304a8b487ee: Status 404 returned error can't find the container with id 084d2d66e1f581d89b945ef59d66ce61b18c43b1cb9a906be551f304a8b487ee Jul 10 20:13:07.269257 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:07.269221 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" event={"ID":"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495","Type":"ContainerStarted","Data":"084d2d66e1f581d89b945ef59d66ce61b18c43b1cb9a906be551f304a8b487ee"} Jul 10 20:13:11.294451 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:11.294408 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" event={"ID":"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495","Type":"ContainerStarted","Data":"1c712b8a40742cad192971a853d8870000dd2c9d604929fa6b57752d4859756c"} Jul 10 20:13:12.300401 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:12.300363 2565 generic.go:356] "Generic (PLEG): container finished" podID="f53333fc-9d3a-4bbc-aca6-5d2d2cb36495" containerID="1c712b8a40742cad192971a853d8870000dd2c9d604929fa6b57752d4859756c" exitCode=0 Jul 10 20:13:12.300907 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:12.300422 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" event={"ID":"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495","Type":"ContainerDied","Data":"1c712b8a40742cad192971a853d8870000dd2c9d604929fa6b57752d4859756c"} Jul 10 20:13:13.306596 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:13.306562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" event={"ID":"f53333fc-9d3a-4bbc-aca6-5d2d2cb36495","Type":"ContainerStarted","Data":"17d14fc69f4cf3551c2764b9f64b08d03fc0155d627b43f2b8dedd53dad81251"} Jul 10 20:13:13.307002 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:13.306640 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:13.329787 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:13.329728 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" podStartSLOduration=2.470028462 podStartE2EDuration="7.329709743s" totalImagesPullingTime="3.798652901s" totalInitContainerRuntime="1.06102838s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:13:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:13:13.327609805 +0000 UTC m=+273.800579132" watchObservedRunningTime="2026-07-10 20:13:13.329709743 +0000 UTC m=+273.802679069" Jul 10 20:13:24.312981 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:24.312946 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5767ffdff4-fv827" Jul 10 20:13:25.424396 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.424355 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf"] Jul 10 20:13:25.432666 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.432641 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.436846 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.436821 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lh8n7\"" Jul 10 20:13:25.436846 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.436835 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 20:13:25.437486 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.437472 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 20:13:25.440204 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.440134 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 20:13:25.440750 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.440730 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf"] Jul 10 20:13:25.535477 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.535436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/921861e7-7e09-47ea-a34f-2e876cbf2f4b-cert\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.535477 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.535476 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/921861e7-7e09-47ea-a34f-2e876cbf2f4b-manager-config\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.535684 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.535505 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrwt5\" (UniqueName: \"kubernetes.io/projected/921861e7-7e09-47ea-a34f-2e876cbf2f4b-kube-api-access-lrwt5\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.535684 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.535626 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/921861e7-7e09-47ea-a34f-2e876cbf2f4b-metrics-cert\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.636397 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.636355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/921861e7-7e09-47ea-a34f-2e876cbf2f4b-metrics-cert\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.636559 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.636421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/921861e7-7e09-47ea-a34f-2e876cbf2f4b-cert\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.636559 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.636450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/921861e7-7e09-47ea-a34f-2e876cbf2f4b-manager-config\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.636559 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.636471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrwt5\" (UniqueName: \"kubernetes.io/projected/921861e7-7e09-47ea-a34f-2e876cbf2f4b-kube-api-access-lrwt5\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.637313 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.637242 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/921861e7-7e09-47ea-a34f-2e876cbf2f4b-manager-config\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.639315 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.639289 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/921861e7-7e09-47ea-a34f-2e876cbf2f4b-metrics-cert\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.639422 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.639289 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/921861e7-7e09-47ea-a34f-2e876cbf2f4b-cert\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.664816 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.664776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrwt5\" (UniqueName: \"kubernetes.io/projected/921861e7-7e09-47ea-a34f-2e876cbf2f4b-kube-api-access-lrwt5\") pod \"lws-controller-manager-7bd7b6bf4b-7lfpf\" (UID: \"921861e7-7e09-47ea-a34f-2e876cbf2f4b\") " pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.743071 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.742982 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:25.902741 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:25.902697 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf"] Jul 10 20:13:25.907029 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:13:25.906998 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921861e7_7e09_47ea_a34f_2e876cbf2f4b.slice/crio-6e004c261fda91823ba9bd10881b2d291a1e151cab4e801c810d46318088bf02 WatchSource:0}: Error finding container 6e004c261fda91823ba9bd10881b2d291a1e151cab4e801c810d46318088bf02: Status 404 returned error can't find the container with id 6e004c261fda91823ba9bd10881b2d291a1e151cab4e801c810d46318088bf02 Jul 10 20:13:26.358841 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.358802 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" event={"ID":"921861e7-7e09-47ea-a34f-2e876cbf2f4b","Type":"ContainerStarted","Data":"6e004c261fda91823ba9bd10881b2d291a1e151cab4e801c810d46318088bf02"} Jul 10 20:13:26.684772 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.684732 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rbztx"] Jul 10 20:13:26.688828 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.688810 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:26.691297 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.691274 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 20:13:26.691438 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.691304 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nmbtr\"" Jul 10 20:13:26.698581 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.698554 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rbztx"] Jul 10 20:13:26.744519 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.744479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad5355c-7671-4040-acae-49bfbfe60002-cert\") pod \"odh-model-controller-7998c67dc4-rbztx\" (UID: \"aad5355c-7671-4040-acae-49bfbfe60002\") " pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:26.744719 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.744544 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rmw\" (UniqueName: \"kubernetes.io/projected/aad5355c-7671-4040-acae-49bfbfe60002-kube-api-access-j6rmw\") pod \"odh-model-controller-7998c67dc4-rbztx\" (UID: \"aad5355c-7671-4040-acae-49bfbfe60002\") " pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:26.845196 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.845159 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rmw\" (UniqueName: \"kubernetes.io/projected/aad5355c-7671-4040-acae-49bfbfe60002-kube-api-access-j6rmw\") pod \"odh-model-controller-7998c67dc4-rbztx\" (UID: \"aad5355c-7671-4040-acae-49bfbfe60002\") " pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:26.845399 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.845246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad5355c-7671-4040-acae-49bfbfe60002-cert\") pod \"odh-model-controller-7998c67dc4-rbztx\" (UID: \"aad5355c-7671-4040-acae-49bfbfe60002\") " pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:26.845494 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:13:26.845391 2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 20:13:26.845494 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:13:26.845482 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad5355c-7671-4040-acae-49bfbfe60002-cert podName:aad5355c-7671-4040-acae-49bfbfe60002 nodeName:}" failed. No retries permitted until 2026-07-10 20:13:27.345459525 +0000 UTC m=+287.818428830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aad5355c-7671-4040-acae-49bfbfe60002-cert") pod "odh-model-controller-7998c67dc4-rbztx" (UID: "aad5355c-7671-4040-acae-49bfbfe60002") : secret "odh-model-controller-webhook-cert" not found Jul 10 20:13:26.856973 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:26.856938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rmw\" (UniqueName: \"kubernetes.io/projected/aad5355c-7671-4040-acae-49bfbfe60002-kube-api-access-j6rmw\") pod \"odh-model-controller-7998c67dc4-rbztx\" (UID: \"aad5355c-7671-4040-acae-49bfbfe60002\") " pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:27.347797 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:27.347759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad5355c-7671-4040-acae-49bfbfe60002-cert\") pod \"odh-model-controller-7998c67dc4-rbztx\" (UID: \"aad5355c-7671-4040-acae-49bfbfe60002\") " pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:27.350769 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:27.350737 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad5355c-7671-4040-acae-49bfbfe60002-cert\") pod \"odh-model-controller-7998c67dc4-rbztx\" (UID: \"aad5355c-7671-4040-acae-49bfbfe60002\") " pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:27.600900 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:27.600825 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:27.744330 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:27.744305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rbztx"] Jul 10 20:13:27.746716 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:13:27.746685 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad5355c_7671_4040_acae_49bfbfe60002.slice/crio-eeec41c15a5c3af5a6a9ea6949aadd76e9dcc91332e8b8c46e7dc8a60ca111c1 WatchSource:0}: Error finding container eeec41c15a5c3af5a6a9ea6949aadd76e9dcc91332e8b8c46e7dc8a60ca111c1: Status 404 returned error can't find the container with id eeec41c15a5c3af5a6a9ea6949aadd76e9dcc91332e8b8c46e7dc8a60ca111c1 Jul 10 20:13:28.369819 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:28.369780 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" event={"ID":"aad5355c-7671-4040-acae-49bfbfe60002","Type":"ContainerStarted","Data":"eeec41c15a5c3af5a6a9ea6949aadd76e9dcc91332e8b8c46e7dc8a60ca111c1"} Jul 10 20:13:28.371588 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:28.371554 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" event={"ID":"921861e7-7e09-47ea-a34f-2e876cbf2f4b","Type":"ContainerStarted","Data":"1d87ce3488902b344c52847c688a7ae74a41dd3004c16726f13ef16b6cd440cc"} Jul 10 20:13:28.371789 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:28.371662 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:28.387154 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:28.387097 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" podStartSLOduration=1.703795387 podStartE2EDuration="3.387076379s" totalImagesPullingTime="1.683280992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:13:28.386653681 +0000 UTC m=+288.859623008" watchObservedRunningTime="2026-07-10 20:13:28.387076379 +0000 UTC m=+288.860045705" Jul 10 20:13:31.386900 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:31.386843 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" event={"ID":"aad5355c-7671-4040-acae-49bfbfe60002","Type":"ContainerStarted","Data":"652c3f08df3a91878cf6eb3ef5bc138e0af9d14bc5edbe8f8fc34ce59f34cba5"} Jul 10 20:13:31.387301 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:31.386969 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:13:31.404964 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:31.404910 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" podStartSLOduration=2.530519125 podStartE2EDuration="5.404891118s" totalImagesPullingTime="2.874371993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:13:31.403947779 +0000 UTC m=+291.876917107" watchObservedRunningTime="2026-07-10 20:13:31.404891118 +0000 UTC m=+291.877860441" Jul 10 20:13:36.847400 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.847363 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-74575f44df-7v8gk"] Jul 10 20:13:36.850373 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.850350 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:36.855438 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.855377 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 20:13:36.855665 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.855639 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 20:13:36.856261 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.856241 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 20:13:36.856514 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.856496 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wphfl\"" Jul 10 20:13:36.856577 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.856505 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 20:13:36.871954 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.871912 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74575f44df-7v8gk"] Jul 10 20:13:36.915322 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.915284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9697b134-546f-4f75-9842-104a491b0e59-tmp\") pod \"kube-auth-proxy-74575f44df-7v8gk\" (UID: \"9697b134-546f-4f75-9842-104a491b0e59\") " pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:36.915493 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.915334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9697b134-546f-4f75-9842-104a491b0e59-tls-certs\") pod \"kube-auth-proxy-74575f44df-7v8gk\" (UID: \"9697b134-546f-4f75-9842-104a491b0e59\") " pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:36.915493 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:36.915360 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bzw7\" (UniqueName: \"kubernetes.io/projected/9697b134-546f-4f75-9842-104a491b0e59-kube-api-access-7bzw7\") pod \"kube-auth-proxy-74575f44df-7v8gk\" (UID: \"9697b134-546f-4f75-9842-104a491b0e59\") " pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:37.015749 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:37.015718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9697b134-546f-4f75-9842-104a491b0e59-tls-certs\") pod \"kube-auth-proxy-74575f44df-7v8gk\" (UID: \"9697b134-546f-4f75-9842-104a491b0e59\") " pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:37.015749 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:37.015753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bzw7\" (UniqueName: \"kubernetes.io/projected/9697b134-546f-4f75-9842-104a491b0e59-kube-api-access-7bzw7\") pod \"kube-auth-proxy-74575f44df-7v8gk\" (UID: \"9697b134-546f-4f75-9842-104a491b0e59\") " pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:37.016039 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:37.015814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9697b134-546f-4f75-9842-104a491b0e59-tmp\") pod \"kube-auth-proxy-74575f44df-7v8gk\" (UID: \"9697b134-546f-4f75-9842-104a491b0e59\") " pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:37.018308 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:37.018277 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9697b134-546f-4f75-9842-104a491b0e59-tmp\") pod \"kube-auth-proxy-74575f44df-7v8gk\" (UID: \"9697b134-546f-4f75-9842-104a491b0e59\") " pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:37.018428 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:37.018413 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9697b134-546f-4f75-9842-104a491b0e59-tls-certs\") pod \"kube-auth-proxy-74575f44df-7v8gk\" (UID: \"9697b134-546f-4f75-9842-104a491b0e59\") " pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:37.033422 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:37.033387 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bzw7\" (UniqueName: \"kubernetes.io/projected/9697b134-546f-4f75-9842-104a491b0e59-kube-api-access-7bzw7\") pod \"kube-auth-proxy-74575f44df-7v8gk\" (UID: \"9697b134-546f-4f75-9842-104a491b0e59\") " pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:37.160294 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:37.160260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" Jul 10 20:13:37.302913 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:37.302885 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74575f44df-7v8gk"] Jul 10 20:13:37.305588 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:13:37.305554 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9697b134_546f_4f75_9842_104a491b0e59.slice/crio-264f4a7be9db7e8e7c3f131f07ee1222f296bec719adf94d5ed1a292c969b1d2 WatchSource:0}: Error finding container 264f4a7be9db7e8e7c3f131f07ee1222f296bec719adf94d5ed1a292c969b1d2: Status 404 returned error can't find the container with id 264f4a7be9db7e8e7c3f131f07ee1222f296bec719adf94d5ed1a292c969b1d2 Jul 10 20:13:37.413416 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:37.413328 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" event={"ID":"9697b134-546f-4f75-9842-104a491b0e59","Type":"ContainerStarted","Data":"264f4a7be9db7e8e7c3f131f07ee1222f296bec719adf94d5ed1a292c969b1d2"} Jul 10 20:13:39.379855 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:39.379816 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bd7b6bf4b-7lfpf" Jul 10 20:13:39.893404 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:39.893378 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:13:39.893592 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:39.893461 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:13:39.900738 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:39.900714 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 20:13:40.646651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:40.646632 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 20:13:41.435361 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:41.435326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" event={"ID":"9697b134-546f-4f75-9842-104a491b0e59","Type":"ContainerStarted","Data":"67220a1696a56d0a65b3e4ae5b3585d38ecf3e6bad35060f2366959af16c8762"} Jul 10 20:13:41.450304 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:41.450252 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-74575f44df-7v8gk" podStartSLOduration=2.113596934 podStartE2EDuration="5.45023523s" totalImagesPullingTime="3.336638296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:13:41.449583113 +0000 UTC m=+301.922552441" watchObservedRunningTime="2026-07-10 20:13:41.45023523 +0000 UTC m=+301.923204556" Jul 10 20:13:42.393440 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:13:42.393407 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rbztx" Jul 10 20:15:01.299011 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.298934 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65cb9d569b-7pldd"] Jul 10 20:15:01.302067 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.302047 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.313096 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.313069 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cb9d569b-7pldd"] Jul 10 20:15:01.458670 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.458624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-console-oauth-config\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.458670 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.458671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-console-serving-cert\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.458920 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.458696 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-trusted-ca-bundle\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.458920 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.458776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-service-ca\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.458920 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.458853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w222\" (UniqueName: \"kubernetes.io/projected/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-kube-api-access-8w222\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.458920 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.458900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-oauth-serving-cert\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.458920 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.458918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-console-config\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.559701 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.559610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-service-ca\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.559701 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.559667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w222\" (UniqueName: \"kubernetes.io/projected/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-kube-api-access-8w222\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.559701 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.559696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-oauth-serving-cert\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.559978 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.559721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-console-config\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.559978 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.559782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-console-oauth-config\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.560074 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.559964 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-console-serving-cert\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.560074 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.560022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-trusted-ca-bundle\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.560528 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.560496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-console-config\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.560656 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.560640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-service-ca\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.561011 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.560984 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-oauth-serving-cert\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.561329 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.561303 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-trusted-ca-bundle\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.562535 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.562505 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-console-oauth-config\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.562693 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.562672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-console-serving-cert\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.576554 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.576521 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w222\" (UniqueName: \"kubernetes.io/projected/6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5-kube-api-access-8w222\") pod \"console-65cb9d569b-7pldd\" (UID: \"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5\") " pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.611417 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.611365 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:01.755382 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.755358 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cb9d569b-7pldd"] Jul 10 20:15:01.757662 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:15:01.757633 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3ae028_f71d_4c14_8c9b_1f3abcf9dfd5.slice/crio-2fde7120ec1c42e8126083f749eee61d0dd74bdad2ea196bfe544e09676bb4c5 WatchSource:0}: Error finding container 2fde7120ec1c42e8126083f749eee61d0dd74bdad2ea196bfe544e09676bb4c5: Status 404 returned error can't find the container with id 2fde7120ec1c42e8126083f749eee61d0dd74bdad2ea196bfe544e09676bb4c5 Jul 10 20:15:01.781650 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:01.781601 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cb9d569b-7pldd" event={"ID":"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5","Type":"ContainerStarted","Data":"2fde7120ec1c42e8126083f749eee61d0dd74bdad2ea196bfe544e09676bb4c5"} Jul 10 20:15:02.787620 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:02.787579 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cb9d569b-7pldd" event={"ID":"6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5","Type":"ContainerStarted","Data":"0e2a04909af1c6ce8f0b01db34be22450ccd3108951a22be140ba05b3baffc81"} Jul 10 20:15:02.806000 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:02.805950 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65cb9d569b-7pldd" podStartSLOduration=1.805931425 podStartE2EDuration="1.805931425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:15:02.804553205 +0000 UTC m=+383.277522530" watchObservedRunningTime="2026-07-10 20:15:02.805931425 +0000 UTC m=+383.278900748" Jul 10 20:15:03.446689 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.446654 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft"] Jul 10 20:15:03.450407 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.450387 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" Jul 10 20:15:03.452473 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.452447 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 20:15:03.453383 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.453365 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zft8h\"" Jul 10 20:15:03.453509 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.453395 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 20:15:03.453509 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.453429 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 20:15:03.461064 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.461037 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft"] Jul 10 20:15:03.577943 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.577901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5ps\" (UniqueName: \"kubernetes.io/projected/d4e5e042-290e-47bb-84af-567017e99f9c-kube-api-access-dw5ps\") pod \"dns-operator-controller-manager-864dd78c4c-pvgft\" (UID: \"d4e5e042-290e-47bb-84af-567017e99f9c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" Jul 10 20:15:03.678910 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.678850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5ps\" (UniqueName: \"kubernetes.io/projected/d4e5e042-290e-47bb-84af-567017e99f9c-kube-api-access-dw5ps\") pod \"dns-operator-controller-manager-864dd78c4c-pvgft\" (UID: \"d4e5e042-290e-47bb-84af-567017e99f9c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" Jul 10 20:15:03.693176 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.693146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5ps\" (UniqueName: \"kubernetes.io/projected/d4e5e042-290e-47bb-84af-567017e99f9c-kube-api-access-dw5ps\") pod \"dns-operator-controller-manager-864dd78c4c-pvgft\" (UID: \"d4e5e042-290e-47bb-84af-567017e99f9c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" Jul 10 20:15:03.762117 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.762020 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" Jul 10 20:15:03.905163 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.905132 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft"] Jul 10 20:15:03.908586 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:15:03.908547 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e5e042_290e_47bb_84af_567017e99f9c.slice/crio-7a6e70b2b4ff358822367756f6169ab567e25574ef2436e0bc7f316c8c9acac0 WatchSource:0}: Error finding container 7a6e70b2b4ff358822367756f6169ab567e25574ef2436e0bc7f316c8c9acac0: Status 404 returned error can't find the container with id 7a6e70b2b4ff358822367756f6169ab567e25574ef2436e0bc7f316c8c9acac0 Jul 10 20:15:03.911179 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:03.911163 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 20:15:04.797218 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:04.797178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" event={"ID":"d4e5e042-290e-47bb-84af-567017e99f9c","Type":"ContainerStarted","Data":"7a6e70b2b4ff358822367756f6169ab567e25574ef2436e0bc7f316c8c9acac0"} Jul 10 20:15:06.807941 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:06.807823 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" event={"ID":"d4e5e042-290e-47bb-84af-567017e99f9c","Type":"ContainerStarted","Data":"d0d961c512da09cf8510882482b9f40a80f262da99ef69f6c79b8fe279984b29"} Jul 10 20:15:06.807941 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:06.807934 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" Jul 10 20:15:06.829320 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:06.829264 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" podStartSLOduration=1.232188532 podStartE2EDuration="3.829248043s" totalImagesPullingTime="2.597059511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:15:06.827024312 +0000 UTC m=+387.299993639" watchObservedRunningTime="2026-07-10 20:15:06.829248043 +0000 UTC m=+387.302217368" Jul 10 20:15:11.611529 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.611485 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:11.611944 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.611622 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:11.616607 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.616583 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:11.705935 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.705898 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw"] Jul 10 20:15:11.709236 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.709212 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:11.712466 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.712443 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z247m\"" Jul 10 20:15:11.723195 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.723166 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw"] Jul 10 20:15:11.753674 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.753639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m5rz\" (UniqueName: \"kubernetes.io/projected/bb7ed47f-0951-4555-be0e-0d4802231acc-kube-api-access-4m5rz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" (UID: \"bb7ed47f-0951-4555-be0e-0d4802231acc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:11.753831 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.753694 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb7ed47f-0951-4555-be0e-0d4802231acc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" (UID: \"bb7ed47f-0951-4555-be0e-0d4802231acc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:11.834333 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.834304 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65cb9d569b-7pldd" Jul 10 20:15:11.854769 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.854725 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5rz\" (UniqueName: \"kubernetes.io/projected/bb7ed47f-0951-4555-be0e-0d4802231acc-kube-api-access-4m5rz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" (UID: \"bb7ed47f-0951-4555-be0e-0d4802231acc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:11.855004 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.854807 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb7ed47f-0951-4555-be0e-0d4802231acc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" (UID: \"bb7ed47f-0951-4555-be0e-0d4802231acc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:11.855337 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.855316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb7ed47f-0951-4555-be0e-0d4802231acc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" (UID: \"bb7ed47f-0951-4555-be0e-0d4802231acc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:11.867562 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.867488 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5rz\" (UniqueName: \"kubernetes.io/projected/bb7ed47f-0951-4555-be0e-0d4802231acc-kube-api-access-4m5rz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" (UID: \"bb7ed47f-0951-4555-be0e-0d4802231acc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:11.878534 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:11.878502 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbf9484b4-mrxjk"] Jul 10 20:15:12.019262 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:12.019222 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:12.163841 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:12.163655 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw"] Jul 10 20:15:12.166962 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:15:12.166930 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ed47f_0951_4555_be0e_0d4802231acc.slice/crio-40d76e1a9e4e800d584e4bd003951f3b5cb5ce023fbfa9b686e7b78b016aeab2 WatchSource:0}: Error finding container 40d76e1a9e4e800d584e4bd003951f3b5cb5ce023fbfa9b686e7b78b016aeab2: Status 404 returned error can't find the container with id 40d76e1a9e4e800d584e4bd003951f3b5cb5ce023fbfa9b686e7b78b016aeab2 Jul 10 20:15:12.836134 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:12.836104 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" event={"ID":"bb7ed47f-0951-4555-be0e-0d4802231acc","Type":"ContainerStarted","Data":"40d76e1a9e4e800d584e4bd003951f3b5cb5ce023fbfa9b686e7b78b016aeab2"} Jul 10 20:15:17.814664 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:17.814628 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pvgft" Jul 10 20:15:19.872341 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:19.872305 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" event={"ID":"bb7ed47f-0951-4555-be0e-0d4802231acc","Type":"ContainerStarted","Data":"0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524"} Jul 10 20:15:19.872730 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:19.872436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:19.891366 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:19.891306 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" podStartSLOduration=1.975972192 podStartE2EDuration="8.891279991s" totalImagesPullingTime="6.915307799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:15:19.889947082 +0000 UTC m=+400.362916408" watchObservedRunningTime="2026-07-10 20:15:19.891279991 +0000 UTC m=+400.364249383" Jul 10 20:15:21.884773 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:21.884735 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8"] Jul 10 20:15:21.888185 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:21.888167 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:21.890784 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:21.890757 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 20:15:21.890991 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:21.890820 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 20:15:21.891150 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:21.891135 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7rtwr\"" Jul 10 20:15:21.896130 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:21.896003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8"] Jul 10 20:15:21.952229 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:21.952196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d96ba105-89ab-4dc3-b139-28ded2c45713-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pzjc8\" (UID: \"d96ba105-89ab-4dc3-b139-28ded2c45713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:21.952414 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:21.952237 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d96ba105-89ab-4dc3-b139-28ded2c45713-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pzjc8\" (UID: \"d96ba105-89ab-4dc3-b139-28ded2c45713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:21.952414 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:21.952257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4vq\" (UniqueName: \"kubernetes.io/projected/d96ba105-89ab-4dc3-b139-28ded2c45713-kube-api-access-tw4vq\") pod \"kuadrant-console-plugin-66b5774985-pzjc8\" (UID: \"d96ba105-89ab-4dc3-b139-28ded2c45713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:22.053571 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:22.053539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d96ba105-89ab-4dc3-b139-28ded2c45713-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pzjc8\" (UID: \"d96ba105-89ab-4dc3-b139-28ded2c45713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:22.053765 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:22.053579 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4vq\" (UniqueName: \"kubernetes.io/projected/d96ba105-89ab-4dc3-b139-28ded2c45713-kube-api-access-tw4vq\") pod \"kuadrant-console-plugin-66b5774985-pzjc8\" (UID: \"d96ba105-89ab-4dc3-b139-28ded2c45713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:22.053765 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:22.053688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d96ba105-89ab-4dc3-b139-28ded2c45713-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pzjc8\" (UID: \"d96ba105-89ab-4dc3-b139-28ded2c45713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:22.054301 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:22.054275 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d96ba105-89ab-4dc3-b139-28ded2c45713-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pzjc8\" (UID: \"d96ba105-89ab-4dc3-b139-28ded2c45713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:22.056400 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:22.056377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d96ba105-89ab-4dc3-b139-28ded2c45713-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pzjc8\" (UID: \"d96ba105-89ab-4dc3-b139-28ded2c45713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:22.061432 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:22.061408 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4vq\" (UniqueName: \"kubernetes.io/projected/d96ba105-89ab-4dc3-b139-28ded2c45713-kube-api-access-tw4vq\") pod \"kuadrant-console-plugin-66b5774985-pzjc8\" (UID: \"d96ba105-89ab-4dc3-b139-28ded2c45713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:22.198960 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:22.198845 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" Jul 10 20:15:22.332694 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:22.332671 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8"] Jul 10 20:15:22.334904 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:15:22.334878 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd96ba105_89ab_4dc3_b139_28ded2c45713.slice/crio-a3f3959bc7f60dd72b1ac6dd6b89bcabed446349e094bc48b121f3b3c6307247 WatchSource:0}: Error finding container a3f3959bc7f60dd72b1ac6dd6b89bcabed446349e094bc48b121f3b3c6307247: Status 404 returned error can't find the container with id a3f3959bc7f60dd72b1ac6dd6b89bcabed446349e094bc48b121f3b3c6307247 Jul 10 20:15:22.886915 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:22.886856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" event={"ID":"d96ba105-89ab-4dc3-b139-28ded2c45713","Type":"ContainerStarted","Data":"a3f3959bc7f60dd72b1ac6dd6b89bcabed446349e094bc48b121f3b3c6307247"} Jul 10 20:15:30.879556 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:30.879525 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:32.741027 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.740989 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw"] Jul 10 20:15:32.741613 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.741206 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" containerName="manager" containerID="cri-o://0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524" gracePeriod=2 Jul 10 20:15:32.756612 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:32.756568 2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"bb7ed47f-0951-4555-be0e-0d4802231acc\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\": pods \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:32.760002 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.759927 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw"] Jul 10 20:15:32.774311 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.774267 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5"] Jul 10 20:15:32.774800 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.774770 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" containerName="manager" Jul 10 20:15:32.774800 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.774795 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" containerName="manager" Jul 10 20:15:32.775018 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.774969 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" containerName="manager" Jul 10 20:15:32.778384 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.778358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:32.780600 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:32.780550 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" is forbidden: User \"system:node:ip-10-0-143-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-108.ec2.internal' and this object" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:32.796486 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.796409 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5"] Jul 10 20:15:32.860660 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.860618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-clvj5\" (UID: \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:32.860888 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.860708 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvk7\" (UniqueName: \"kubernetes.io/projected/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-kube-api-access-vsvk7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-clvj5\" (UID: \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:32.961332 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.961292 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-clvj5\" (UID: \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:32.961528 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.961374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvk7\" (UniqueName: \"kubernetes.io/projected/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-kube-api-access-vsvk7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-clvj5\" (UID: \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:32.961694 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.961672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-clvj5\" (UID: \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:32.973467 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:32.973433 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvk7\" (UniqueName: \"kubernetes.io/projected/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-kube-api-access-vsvk7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-clvj5\" (UID: \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:33.119487 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:33.119394 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:36.900734 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:36.900692 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fbf9484b4-mrxjk" podUID="8c24bdf5-18e8-4186-8544-d6019620a1d9" containerName="console" containerID="cri-o://8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e" gracePeriod=15 Jul 10 20:15:45.848809 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:45.848768 2565 patch_prober.go:28] interesting pod/console-6fbf9484b4-mrxjk container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused" start-of-body= Jul 10 20:15:45.849269 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:45.848855 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6fbf9484b4-mrxjk" podUID="8c24bdf5-18e8-4186-8544-d6019620a1d9" containerName="console" probeResult="failure" output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused" Jul 10 20:15:46.933799 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:46.933770 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:15:46.938064 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:46.937662 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:46.939112 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:46.939089 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5"] Jul 10 20:15:46.941752 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:15:46.941723 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c70251_5d87_42d4_8f2d_6b7f2cc2996e.slice/crio-30e2b173cccd6c69704b887eb67375792471ac03a1e3114a93e9a69ff0c80ce7 WatchSource:0}: Error finding container 30e2b173cccd6c69704b887eb67375792471ac03a1e3114a93e9a69ff0c80ce7: Status 404 returned error can't find the container with id 30e2b173cccd6c69704b887eb67375792471ac03a1e3114a93e9a69ff0c80ce7 Jul 10 20:15:46.952650 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:46.952598 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" is forbidden: User \"system:node:ip-10-0-143-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-108.ec2.internal' and this object" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:47.014735 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.014701 2565 generic.go:356] "Generic (PLEG): container finished" podID="8c24bdf5-18e8-4186-8544-d6019620a1d9" containerID="8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e" exitCode=0 Jul 10 20:15:47.014936 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.014746 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbf9484b4-mrxjk" Jul 10 20:15:47.014936 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.014740 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbf9484b4-mrxjk" event={"ID":"8c24bdf5-18e8-4186-8544-d6019620a1d9","Type":"ContainerDied","Data":"8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e"} Jul 10 20:15:47.014936 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.014856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbf9484b4-mrxjk" event={"ID":"8c24bdf5-18e8-4186-8544-d6019620a1d9","Type":"ContainerDied","Data":"672a800a87e4334ab2472005c27c1603cf97c60599674d2aae94f3737dd2b0d3"} Jul 10 20:15:47.014936 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.014904 2565 scope.go:117] "RemoveContainer" containerID="8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e" Jul 10 20:15:47.016278 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.016256 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb7ed47f-0951-4555-be0e-0d4802231acc" containerID="0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524" exitCode=0 Jul 10 20:15:47.016480 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.016465 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:47.018083 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.018055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" event={"ID":"d96ba105-89ab-4dc3-b139-28ded2c45713","Type":"ContainerStarted","Data":"c59e56c628fa402f91d3be8a446807fdd893bc127be2090affb74e44dafc7265"} Jul 10 20:15:47.019713 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.019689 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" event={"ID":"22c70251-5d87-42d4-8f2d-6b7f2cc2996e","Type":"ContainerStarted","Data":"126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03"} Jul 10 20:15:47.019838 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.019738 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" event={"ID":"22c70251-5d87-42d4-8f2d-6b7f2cc2996e","Type":"ContainerStarted","Data":"30e2b173cccd6c69704b887eb67375792471ac03a1e3114a93e9a69ff0c80ce7"} Jul 10 20:15:47.019838 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.019769 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:47.020111 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:47.020078 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" is forbidden: User \"system:node:ip-10-0-143-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-108.ec2.internal' and this object" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:47.024132 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.024109 2565 scope.go:117] "RemoveContainer" containerID="8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e" Jul 10 20:15:47.024388 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:47.024371 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e\": container with ID starting with 8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e not found: ID does not exist" containerID="8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e" Jul 10 20:15:47.024434 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.024396 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e"} err="failed to get container status \"8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e\": rpc error: code = NotFound desc = could not find container \"8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e\": container with ID starting with 8736987f85ff3b37e2b174fe14427f59f1b32195b8024c037ca98d071ef79f9e not found: ID does not exist" Jul 10 20:15:47.024434 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.024411 2565 scope.go:117] "RemoveContainer" containerID="0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524" Jul 10 20:15:47.032429 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.032411 2565 scope.go:117] "RemoveContainer" containerID="0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524" Jul 10 20:15:47.032731 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:47.032701 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524\": container with ID starting with 0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524 not found: ID does not exist" containerID="0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524" Jul 10 20:15:47.032823 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.032727 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524"} err="failed to get container status \"0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524\": rpc error: code = NotFound desc = could not find container \"0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524\": container with ID starting with 0ba8e297ea194b974ccf6811e079dea6f69ed62f1f6b1c49071d8c54c1d1a524 not found: ID does not exist" Jul 10 20:15:47.034899 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.034829 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pzjc8" podStartSLOduration=1.537098309 podStartE2EDuration="26.034812648s" totalImagesPullingTime="24.497714339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:15:47.033693877 +0000 UTC m=+427.506663202" watchObservedRunningTime="2026-07-10 20:15:47.034812648 +0000 UTC m=+427.507781977" Jul 10 20:15:47.054082 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.054017 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" podStartSLOduration=15.053993307 podStartE2EDuration="15.053993307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:15:47.052592708 +0000 UTC m=+427.525562033" watchObservedRunningTime="2026-07-10 20:15:47.053993307 +0000 UTC m=+427.526962634" Jul 10 20:15:47.054358 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:47.054329 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" is forbidden: User \"system:node:ip-10-0-143-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-108.ec2.internal' and this object" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:47.102576 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.102533 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-service-ca\") pod \"8c24bdf5-18e8-4186-8544-d6019620a1d9\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " Jul 10 20:15:47.102880 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.102841 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "8c24bdf5-18e8-4186-8544-d6019620a1d9" (UID: "8c24bdf5-18e8-4186-8544-d6019620a1d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:15:47.103106 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103085 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-trusted-ca-bundle\") pod \"8c24bdf5-18e8-4186-8544-d6019620a1d9\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " Jul 10 20:15:47.103147 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103117 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-config\") pod \"8c24bdf5-18e8-4186-8544-d6019620a1d9\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " Jul 10 20:15:47.103147 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103138 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m5rz\" (UniqueName: \"kubernetes.io/projected/bb7ed47f-0951-4555-be0e-0d4802231acc-kube-api-access-4m5rz\") pod \"bb7ed47f-0951-4555-be0e-0d4802231acc\" (UID: \"bb7ed47f-0951-4555-be0e-0d4802231acc\") " Jul 10 20:15:47.103250 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103156 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-oauth-config\") pod \"8c24bdf5-18e8-4186-8544-d6019620a1d9\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " Jul 10 20:15:47.103250 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103173 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-oauth-serving-cert\") pod \"8c24bdf5-18e8-4186-8544-d6019620a1d9\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " Jul 10 20:15:47.103250 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103203 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb7ed47f-0951-4555-be0e-0d4802231acc-extensions-socket-volume\") pod \"bb7ed47f-0951-4555-be0e-0d4802231acc\" (UID: \"bb7ed47f-0951-4555-be0e-0d4802231acc\") " Jul 10 20:15:47.103250 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103226 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-serving-cert\") pod \"8c24bdf5-18e8-4186-8544-d6019620a1d9\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " Jul 10 20:15:47.103488 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103253 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2pdb\" (UniqueName: \"kubernetes.io/projected/8c24bdf5-18e8-4186-8544-d6019620a1d9-kube-api-access-k2pdb\") pod \"8c24bdf5-18e8-4186-8544-d6019620a1d9\" (UID: \"8c24bdf5-18e8-4186-8544-d6019620a1d9\") " Jul 10 20:15:47.103488 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103450 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-config" (OuterVolumeSpecName: "console-config") pod "8c24bdf5-18e8-4186-8544-d6019620a1d9" (UID: "8c24bdf5-18e8-4186-8544-d6019620a1d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:15:47.103597 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103559 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:47.103597 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103567 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8c24bdf5-18e8-4186-8544-d6019620a1d9" (UID: "8c24bdf5-18e8-4186-8544-d6019620a1d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:15:47.103597 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103579 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-service-ca\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:47.103757 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103729 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7ed47f-0951-4555-be0e-0d4802231acc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bb7ed47f-0951-4555-be0e-0d4802231acc" (UID: "bb7ed47f-0951-4555-be0e-0d4802231acc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:15:47.103971 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.103841 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8c24bdf5-18e8-4186-8544-d6019620a1d9" (UID: "8c24bdf5-18e8-4186-8544-d6019620a1d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 20:15:47.105623 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.105594 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8c24bdf5-18e8-4186-8544-d6019620a1d9" (UID: "8c24bdf5-18e8-4186-8544-d6019620a1d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:15:47.105623 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.105605 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7ed47f-0951-4555-be0e-0d4802231acc-kube-api-access-4m5rz" (OuterVolumeSpecName: "kube-api-access-4m5rz") pod "bb7ed47f-0951-4555-be0e-0d4802231acc" (UID: "bb7ed47f-0951-4555-be0e-0d4802231acc"). InnerVolumeSpecName "kube-api-access-4m5rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:15:47.105829 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.105809 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8c24bdf5-18e8-4186-8544-d6019620a1d9" (UID: "8c24bdf5-18e8-4186-8544-d6019620a1d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:15:47.105987 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.105969 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c24bdf5-18e8-4186-8544-d6019620a1d9-kube-api-access-k2pdb" (OuterVolumeSpecName: "kube-api-access-k2pdb") pod "8c24bdf5-18e8-4186-8544-d6019620a1d9" (UID: "8c24bdf5-18e8-4186-8544-d6019620a1d9"). InnerVolumeSpecName "kube-api-access-k2pdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:15:47.204358 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.204316 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-trusted-ca-bundle\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:47.204358 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.204349 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m5rz\" (UniqueName: \"kubernetes.io/projected/bb7ed47f-0951-4555-be0e-0d4802231acc-kube-api-access-4m5rz\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:47.204358 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.204359 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-oauth-config\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:47.204358 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.204367 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c24bdf5-18e8-4186-8544-d6019620a1d9-oauth-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:47.204617 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.204377 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb7ed47f-0951-4555-be0e-0d4802231acc-extensions-socket-volume\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:47.204617 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.204385 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c24bdf5-18e8-4186-8544-d6019620a1d9-console-serving-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:47.204617 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.204393 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2pdb\" (UniqueName: \"kubernetes.io/projected/8c24bdf5-18e8-4186-8544-d6019620a1d9-kube-api-access-k2pdb\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:47.340106 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.340061 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbf9484b4-mrxjk"] Jul 10 20:15:47.343457 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:47.343414 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" is forbidden: User \"system:node:ip-10-0-143-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-108.ec2.internal' and this object" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:47.343986 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:47.343963 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fbf9484b4-mrxjk"] Jul 10 20:15:47.345104 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:47.345070 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tg5mw\" is forbidden: User \"system:node:ip-10-0-143-108.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-108.ec2.internal' and this object" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tg5mw" Jul 10 20:15:48.040262 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.040185 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c24bdf5-18e8-4186-8544-d6019620a1d9" path="/var/lib/kubelet/pods/8c24bdf5-18e8-4186-8544-d6019620a1d9/volumes" Jul 10 20:15:48.040599 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.040535 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb7ed47f-0951-4555-be0e-0d4802231acc" path="/var/lib/kubelet/pods/bb7ed47f-0951-4555-be0e-0d4802231acc/volumes" Jul 10 20:15:48.464944 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.464907 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5"] Jul 10 20:15:48.688602 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.688561 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p"] Jul 10 20:15:48.689009 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.688990 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c24bdf5-18e8-4186-8544-d6019620a1d9" containerName="console" Jul 10 20:15:48.689009 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.689011 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c24bdf5-18e8-4186-8544-d6019620a1d9" containerName="console" Jul 10 20:15:48.689135 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.689095 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c24bdf5-18e8-4186-8544-d6019620a1d9" containerName="console" Jul 10 20:15:48.692170 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.692144 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:15:48.705530 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.705500 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p"] Jul 10 20:15:48.716086 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.715995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98842af1-aacd-4334-a5e5-b26e486a86ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g6v6p\" (UID: \"98842af1-aacd-4334-a5e5-b26e486a86ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:15:48.716333 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.716311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whppc\" (UniqueName: \"kubernetes.io/projected/98842af1-aacd-4334-a5e5-b26e486a86ee-kube-api-access-whppc\") pod \"kuadrant-operator-controller-manager-86878c769f-g6v6p\" (UID: \"98842af1-aacd-4334-a5e5-b26e486a86ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:15:48.817438 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.817381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98842af1-aacd-4334-a5e5-b26e486a86ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g6v6p\" (UID: \"98842af1-aacd-4334-a5e5-b26e486a86ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:15:48.817635 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.817470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whppc\" (UniqueName: \"kubernetes.io/projected/98842af1-aacd-4334-a5e5-b26e486a86ee-kube-api-access-whppc\") pod \"kuadrant-operator-controller-manager-86878c769f-g6v6p\" (UID: \"98842af1-aacd-4334-a5e5-b26e486a86ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:15:48.817811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.817790 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98842af1-aacd-4334-a5e5-b26e486a86ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g6v6p\" (UID: \"98842af1-aacd-4334-a5e5-b26e486a86ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:15:48.825350 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:48.825317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whppc\" (UniqueName: \"kubernetes.io/projected/98842af1-aacd-4334-a5e5-b26e486a86ee-kube-api-access-whppc\") pod \"kuadrant-operator-controller-manager-86878c769f-g6v6p\" (UID: \"98842af1-aacd-4334-a5e5-b26e486a86ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:15:49.002065 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.001970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:15:49.030297 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.030254 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" podUID="22c70251-5d87-42d4-8f2d-6b7f2cc2996e" containerName="manager" containerID="cri-o://126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03" gracePeriod=10 Jul 10 20:15:49.145367 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.145340 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p"] Jul 10 20:15:49.148141 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:15:49.148105 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98842af1_aacd_4334_a5e5_b26e486a86ee.slice/crio-9297da4e8f80a374b3e2b62211453ed38e91af6eee82053ed5017f278dcb3595 WatchSource:0}: Error finding container 9297da4e8f80a374b3e2b62211453ed38e91af6eee82053ed5017f278dcb3595: Status 404 returned error can't find the container with id 9297da4e8f80a374b3e2b62211453ed38e91af6eee82053ed5017f278dcb3595 Jul 10 20:15:49.303073 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.303047 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:49.321277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.321244 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsvk7\" (UniqueName: \"kubernetes.io/projected/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-kube-api-access-vsvk7\") pod \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\" (UID: \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\") " Jul 10 20:15:49.321401 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.321340 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-extensions-socket-volume\") pod \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\" (UID: \"22c70251-5d87-42d4-8f2d-6b7f2cc2996e\") " Jul 10 20:15:49.321817 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.321781 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "22c70251-5d87-42d4-8f2d-6b7f2cc2996e" (UID: "22c70251-5d87-42d4-8f2d-6b7f2cc2996e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:15:49.323896 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.323839 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-kube-api-access-vsvk7" (OuterVolumeSpecName: "kube-api-access-vsvk7") pod "22c70251-5d87-42d4-8f2d-6b7f2cc2996e" (UID: "22c70251-5d87-42d4-8f2d-6b7f2cc2996e"). InnerVolumeSpecName "kube-api-access-vsvk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:15:49.422202 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.422163 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-extensions-socket-volume\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:49.422202 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:49.422196 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsvk7\" (UniqueName: \"kubernetes.io/projected/22c70251-5d87-42d4-8f2d-6b7f2cc2996e-kube-api-access-vsvk7\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:15:50.038551 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.038512 2565 generic.go:356] "Generic (PLEG): container finished" podID="22c70251-5d87-42d4-8f2d-6b7f2cc2996e" containerID="126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03" exitCode=0 Jul 10 20:15:50.038741 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.038599 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" Jul 10 20:15:50.040900 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.040847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" event={"ID":"98842af1-aacd-4334-a5e5-b26e486a86ee","Type":"ContainerStarted","Data":"db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e"} Jul 10 20:15:50.041075 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.040920 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:15:50.041075 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.040943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" event={"ID":"98842af1-aacd-4334-a5e5-b26e486a86ee","Type":"ContainerStarted","Data":"9297da4e8f80a374b3e2b62211453ed38e91af6eee82053ed5017f278dcb3595"} Jul 10 20:15:50.041075 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.040959 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" event={"ID":"22c70251-5d87-42d4-8f2d-6b7f2cc2996e","Type":"ContainerDied","Data":"126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03"} Jul 10 20:15:50.041075 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.040976 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5" event={"ID":"22c70251-5d87-42d4-8f2d-6b7f2cc2996e","Type":"ContainerDied","Data":"30e2b173cccd6c69704b887eb67375792471ac03a1e3114a93e9a69ff0c80ce7"} Jul 10 20:15:50.041075 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.041000 2565 scope.go:117] "RemoveContainer" containerID="126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03" Jul 10 20:15:50.051555 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.051535 2565 scope.go:117] "RemoveContainer" containerID="126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03" Jul 10 20:15:50.051846 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:15:50.051820 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03\": container with ID starting with 126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03 not found: ID does not exist" containerID="126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03" Jul 10 20:15:50.051964 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.051882 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03"} err="failed to get container status \"126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03\": rpc error: code = NotFound desc = could not find container \"126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03\": container with ID starting with 126eec498d853db720196624674ff8b1ba45499a9bb01d4c06138fc00cec4d03 not found: ID does not exist" Jul 10 20:15:50.064133 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.064076 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" podStartSLOduration=2.064058923 podStartE2EDuration="2.064058923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:15:50.060609454 +0000 UTC m=+430.533578781" watchObservedRunningTime="2026-07-10 20:15:50.064058923 +0000 UTC m=+430.537028248" Jul 10 20:15:50.082536 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.082449 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5"] Jul 10 20:15:50.091605 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:50.091565 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-clvj5"] Jul 10 20:15:52.040985 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:15:52.040948 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c70251-5d87-42d4-8f2d-6b7f2cc2996e" path="/var/lib/kubelet/pods/22c70251-5d87-42d4-8f2d-6b7f2cc2996e/volumes" Jul 10 20:16:01.047395 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:01.047303 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:16:21.581331 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.581294 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-xpjwc"] Jul 10 20:16:21.581801 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.581635 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c70251-5d87-42d4-8f2d-6b7f2cc2996e" containerName="manager" Jul 10 20:16:21.581801 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.581646 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c70251-5d87-42d4-8f2d-6b7f2cc2996e" containerName="manager" Jul 10 20:16:21.581801 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.581717 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c70251-5d87-42d4-8f2d-6b7f2cc2996e" containerName="manager" Jul 10 20:16:21.628603 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.628570 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xpjwc"] Jul 10 20:16:21.628771 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.628661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xpjwc" Jul 10 20:16:21.630876 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.630836 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cckdm\"" Jul 10 20:16:21.710380 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.710341 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbtj\" (UniqueName: \"kubernetes.io/projected/e8163b61-1054-4bd6-84fa-b4926c654b1f-kube-api-access-ljbtj\") pod \"authorino-58cd955675-xpjwc\" (UID: \"e8163b61-1054-4bd6-84fa-b4926c654b1f\") " pod="kuadrant-system/authorino-58cd955675-xpjwc" Jul 10 20:16:21.811463 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.811430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbtj\" (UniqueName: \"kubernetes.io/projected/e8163b61-1054-4bd6-84fa-b4926c654b1f-kube-api-access-ljbtj\") pod \"authorino-58cd955675-xpjwc\" (UID: \"e8163b61-1054-4bd6-84fa-b4926c654b1f\") " pod="kuadrant-system/authorino-58cd955675-xpjwc" Jul 10 20:16:21.819256 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.819221 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbtj\" (UniqueName: \"kubernetes.io/projected/e8163b61-1054-4bd6-84fa-b4926c654b1f-kube-api-access-ljbtj\") pod \"authorino-58cd955675-xpjwc\" (UID: \"e8163b61-1054-4bd6-84fa-b4926c654b1f\") " pod="kuadrant-system/authorino-58cd955675-xpjwc" Jul 10 20:16:21.938538 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:21.938511 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xpjwc" Jul 10 20:16:22.083154 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:22.083122 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xpjwc"] Jul 10 20:16:22.085680 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:16:22.085636 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8163b61_1054_4bd6_84fa_b4926c654b1f.slice/crio-41846a514d0b15cc3075c1a8e7642d5f85f7bc455ce78d3b487c6110bedc7ed1 WatchSource:0}: Error finding container 41846a514d0b15cc3075c1a8e7642d5f85f7bc455ce78d3b487c6110bedc7ed1: Status 404 returned error can't find the container with id 41846a514d0b15cc3075c1a8e7642d5f85f7bc455ce78d3b487c6110bedc7ed1 Jul 10 20:16:22.194685 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:22.194592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xpjwc" event={"ID":"e8163b61-1054-4bd6-84fa-b4926c654b1f","Type":"ContainerStarted","Data":"41846a514d0b15cc3075c1a8e7642d5f85f7bc455ce78d3b487c6110bedc7ed1"} Jul 10 20:16:26.218501 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.218459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xpjwc" event={"ID":"e8163b61-1054-4bd6-84fa-b4926c654b1f","Type":"ContainerStarted","Data":"d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663"} Jul 10 20:16:26.231882 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.231812 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-xpjwc" podStartSLOduration=1.967322944 podStartE2EDuration="5.231794177s" totalImagesPullingTime="3.264471233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:16:26.231076163 +0000 UTC m=+466.704045492" watchObservedRunningTime="2026-07-10 20:16:26.231794177 +0000 UTC m=+466.704763502" Jul 10 20:16:26.448849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.448814 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-8fp2n"] Jul 10 20:16:26.452265 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.452240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8fp2n" Jul 10 20:16:26.458557 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.458529 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8fp2n"] Jul 10 20:16:26.555366 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.555271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247zh\" (UniqueName: \"kubernetes.io/projected/6edc5121-cce3-4a41-8be2-b8289fe14851-kube-api-access-247zh\") pod \"authorino-75d556bd77-8fp2n\" (UID: \"6edc5121-cce3-4a41-8be2-b8289fe14851\") " pod="kuadrant-system/authorino-75d556bd77-8fp2n" Jul 10 20:16:26.656163 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.656123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247zh\" (UniqueName: \"kubernetes.io/projected/6edc5121-cce3-4a41-8be2-b8289fe14851-kube-api-access-247zh\") pod \"authorino-75d556bd77-8fp2n\" (UID: \"6edc5121-cce3-4a41-8be2-b8289fe14851\") " pod="kuadrant-system/authorino-75d556bd77-8fp2n" Jul 10 20:16:26.664548 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.664520 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-247zh\" (UniqueName: \"kubernetes.io/projected/6edc5121-cce3-4a41-8be2-b8289fe14851-kube-api-access-247zh\") pod \"authorino-75d556bd77-8fp2n\" (UID: \"6edc5121-cce3-4a41-8be2-b8289fe14851\") " pod="kuadrant-system/authorino-75d556bd77-8fp2n" Jul 10 20:16:26.762342 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.762305 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8fp2n" Jul 10 20:16:26.897927 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:26.897893 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8fp2n"] Jul 10 20:16:26.900538 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:16:26.900511 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6edc5121_cce3_4a41_8be2_b8289fe14851.slice/crio-a0df1eb1e34012963efdf68b8cac8f817ad4d89e9e2fa531fdb6df7745d1fc48 WatchSource:0}: Error finding container a0df1eb1e34012963efdf68b8cac8f817ad4d89e9e2fa531fdb6df7745d1fc48: Status 404 returned error can't find the container with id a0df1eb1e34012963efdf68b8cac8f817ad4d89e9e2fa531fdb6df7745d1fc48 Jul 10 20:16:27.225400 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:27.225371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8fp2n" event={"ID":"6edc5121-cce3-4a41-8be2-b8289fe14851","Type":"ContainerStarted","Data":"a0df1eb1e34012963efdf68b8cac8f817ad4d89e9e2fa531fdb6df7745d1fc48"} Jul 10 20:16:28.231462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:28.231422 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8fp2n" event={"ID":"6edc5121-cce3-4a41-8be2-b8289fe14851","Type":"ContainerStarted","Data":"57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f"} Jul 10 20:16:28.246066 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:28.246013 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-8fp2n" podStartSLOduration=1.9235390030000001 podStartE2EDuration="2.245987642s" totalImagesPullingTime="322.448639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:16:28.244375956 +0000 UTC m=+468.717345289" watchObservedRunningTime="2026-07-10 20:16:28.245987642 +0000 UTC m=+468.718956968" Jul 10 20:16:28.268037 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:28.268000 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xpjwc"] Jul 10 20:16:28.268219 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:28.268188 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-xpjwc" podUID="e8163b61-1054-4bd6-84fa-b4926c654b1f" containerName="authorino" containerID="cri-o://d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663" gracePeriod=30 Jul 10 20:16:28.525953 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:28.525924 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xpjwc" Jul 10 20:16:28.675600 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:28.675554 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljbtj\" (UniqueName: \"kubernetes.io/projected/e8163b61-1054-4bd6-84fa-b4926c654b1f-kube-api-access-ljbtj\") pod \"e8163b61-1054-4bd6-84fa-b4926c654b1f\" (UID: \"e8163b61-1054-4bd6-84fa-b4926c654b1f\") " Jul 10 20:16:28.677831 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:28.677799 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8163b61-1054-4bd6-84fa-b4926c654b1f-kube-api-access-ljbtj" (OuterVolumeSpecName: "kube-api-access-ljbtj") pod "e8163b61-1054-4bd6-84fa-b4926c654b1f" (UID: "e8163b61-1054-4bd6-84fa-b4926c654b1f"). InnerVolumeSpecName "kube-api-access-ljbtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:16:28.777095 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:28.777004 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljbtj\" (UniqueName: \"kubernetes.io/projected/e8163b61-1054-4bd6-84fa-b4926c654b1f-kube-api-access-ljbtj\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:16:29.236839 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:29.236798 2565 generic.go:356] "Generic (PLEG): container finished" podID="e8163b61-1054-4bd6-84fa-b4926c654b1f" containerID="d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663" exitCode=0 Jul 10 20:16:29.236839 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:29.236837 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xpjwc" Jul 10 20:16:29.237321 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:29.236890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xpjwc" event={"ID":"e8163b61-1054-4bd6-84fa-b4926c654b1f","Type":"ContainerDied","Data":"d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663"} Jul 10 20:16:29.237321 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:29.236937 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xpjwc" event={"ID":"e8163b61-1054-4bd6-84fa-b4926c654b1f","Type":"ContainerDied","Data":"41846a514d0b15cc3075c1a8e7642d5f85f7bc455ce78d3b487c6110bedc7ed1"} Jul 10 20:16:29.237321 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:29.236955 2565 scope.go:117] "RemoveContainer" containerID="d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663" Jul 10 20:16:29.245633 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:29.245614 2565 scope.go:117] "RemoveContainer" containerID="d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663" Jul 10 20:16:29.245972 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:16:29.245952 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663\": container with ID starting with d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663 not found: ID does not exist" containerID="d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663" Jul 10 20:16:29.246044 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:29.245981 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663"} err="failed to get container status \"d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663\": rpc error: code = NotFound desc = could not find container \"d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663\": container with ID starting with d1e7a7440b22e99fbdfdad7c84be22cca337a3b689cfef320991167c1ce7b663 not found: ID does not exist" Jul 10 20:16:29.255702 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:29.255673 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xpjwc"] Jul 10 20:16:29.259645 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:29.259621 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-xpjwc"] Jul 10 20:16:30.042117 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:30.042077 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8163b61-1054-4bd6-84fa-b4926c654b1f" path="/var/lib/kubelet/pods/e8163b61-1054-4bd6-84fa-b4926c654b1f/volumes" Jul 10 20:16:52.320111 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.320071 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-68ll2"] Jul 10 20:16:52.320536 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.320406 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8163b61-1054-4bd6-84fa-b4926c654b1f" containerName="authorino" Jul 10 20:16:52.320536 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.320418 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8163b61-1054-4bd6-84fa-b4926c654b1f" containerName="authorino" Jul 10 20:16:52.320536 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.320479 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8163b61-1054-4bd6-84fa-b4926c654b1f" containerName="authorino" Jul 10 20:16:52.324436 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.324412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-68ll2" Jul 10 20:16:52.328948 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.328919 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-68ll2"] Jul 10 20:16:52.377606 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.377564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqxt9\" (UniqueName: \"kubernetes.io/projected/87a01125-6085-4468-bf2e-523adaa9055e-kube-api-access-xqxt9\") pod \"authorino-5d4b67c446-68ll2\" (UID: \"87a01125-6085-4468-bf2e-523adaa9055e\") " pod="kuadrant-system/authorino-5d4b67c446-68ll2" Jul 10 20:16:52.478436 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.478387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqxt9\" (UniqueName: \"kubernetes.io/projected/87a01125-6085-4468-bf2e-523adaa9055e-kube-api-access-xqxt9\") pod \"authorino-5d4b67c446-68ll2\" (UID: \"87a01125-6085-4468-bf2e-523adaa9055e\") " pod="kuadrant-system/authorino-5d4b67c446-68ll2" Jul 10 20:16:52.486984 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.486954 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqxt9\" (UniqueName: \"kubernetes.io/projected/87a01125-6085-4468-bf2e-523adaa9055e-kube-api-access-xqxt9\") pod \"authorino-5d4b67c446-68ll2\" (UID: \"87a01125-6085-4468-bf2e-523adaa9055e\") " pod="kuadrant-system/authorino-5d4b67c446-68ll2" Jul 10 20:16:52.551377 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.551341 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-68ll2"] Jul 10 20:16:52.551643 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.551630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-68ll2" Jul 10 20:16:52.571690 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.571613 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74d7d9c75f-hs4p8"] Jul 10 20:16:52.577177 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.577149 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" Jul 10 20:16:52.583008 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.582979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74d7d9c75f-hs4p8"] Jul 10 20:16:52.680671 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.680628 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsc7\" (UniqueName: \"kubernetes.io/projected/ff974267-84bc-4c76-8da7-6196f8005911-kube-api-access-lmsc7\") pod \"authorino-74d7d9c75f-hs4p8\" (UID: \"ff974267-84bc-4c76-8da7-6196f8005911\") " pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" Jul 10 20:16:52.690790 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.690763 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-68ll2"] Jul 10 20:16:52.693608 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:16:52.693577 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87a01125_6085_4468_bf2e_523adaa9055e.slice/crio-ba76fe34d27cf6c44c147efbbaf97b5207b6c6cbecbe34994881331560776c90 WatchSource:0}: Error finding container ba76fe34d27cf6c44c147efbbaf97b5207b6c6cbecbe34994881331560776c90: Status 404 returned error can't find the container with id ba76fe34d27cf6c44c147efbbaf97b5207b6c6cbecbe34994881331560776c90 Jul 10 20:16:52.781842 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.781802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsc7\" (UniqueName: \"kubernetes.io/projected/ff974267-84bc-4c76-8da7-6196f8005911-kube-api-access-lmsc7\") pod \"authorino-74d7d9c75f-hs4p8\" (UID: \"ff974267-84bc-4c76-8da7-6196f8005911\") " pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" Jul 10 20:16:52.791422 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.791395 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsc7\" (UniqueName: \"kubernetes.io/projected/ff974267-84bc-4c76-8da7-6196f8005911-kube-api-access-lmsc7\") pod \"authorino-74d7d9c75f-hs4p8\" (UID: \"ff974267-84bc-4c76-8da7-6196f8005911\") " pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" Jul 10 20:16:52.890524 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.890492 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" Jul 10 20:16:52.997782 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:52.997747 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74d7d9c75f-hs4p8"] Jul 10 20:16:53.023194 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.023162 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cbb87d59-lcrw8"] Jul 10 20:16:53.027502 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.027467 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:16:53.029206 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.029179 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74d7d9c75f-hs4p8"] Jul 10 20:16:53.029672 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.029655 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 20:16:53.032350 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:16:53.032325 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff974267_84bc_4c76_8da7_6196f8005911.slice/crio-e54f5e7d0247f0815eeaef46d93b8f4810dac5238f46480f5381cd91b1ad8d73 WatchSource:0}: Error finding container e54f5e7d0247f0815eeaef46d93b8f4810dac5238f46480f5381cd91b1ad8d73: Status 404 returned error can't find the container with id e54f5e7d0247f0815eeaef46d93b8f4810dac5238f46480f5381cd91b1ad8d73 Jul 10 20:16:53.036650 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.036621 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cbb87d59-lcrw8"] Jul 10 20:16:53.084242 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.084201 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13d21514-c497-4de4-a876-2621c35bc0d8-tls-cert\") pod \"authorino-6cbb87d59-lcrw8\" (UID: \"13d21514-c497-4de4-a876-2621c35bc0d8\") " pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:16:53.084432 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.084271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6ql\" (UniqueName: \"kubernetes.io/projected/13d21514-c497-4de4-a876-2621c35bc0d8-kube-api-access-vt6ql\") pod \"authorino-6cbb87d59-lcrw8\" (UID: \"13d21514-c497-4de4-a876-2621c35bc0d8\") " pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:16:53.184738 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.184713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13d21514-c497-4de4-a876-2621c35bc0d8-tls-cert\") pod \"authorino-6cbb87d59-lcrw8\" (UID: \"13d21514-c497-4de4-a876-2621c35bc0d8\") " pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:16:53.184893 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.184751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6ql\" (UniqueName: \"kubernetes.io/projected/13d21514-c497-4de4-a876-2621c35bc0d8-kube-api-access-vt6ql\") pod \"authorino-6cbb87d59-lcrw8\" (UID: \"13d21514-c497-4de4-a876-2621c35bc0d8\") " pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:16:53.187232 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.187209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13d21514-c497-4de4-a876-2621c35bc0d8-tls-cert\") pod \"authorino-6cbb87d59-lcrw8\" (UID: \"13d21514-c497-4de4-a876-2621c35bc0d8\") " pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:16:53.192341 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.192310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6ql\" (UniqueName: \"kubernetes.io/projected/13d21514-c497-4de4-a876-2621c35bc0d8-kube-api-access-vt6ql\") pod \"authorino-6cbb87d59-lcrw8\" (UID: \"13d21514-c497-4de4-a876-2621c35bc0d8\") " pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:16:53.339533 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.339495 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:16:53.350107 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.350068 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" event={"ID":"ff974267-84bc-4c76-8da7-6196f8005911","Type":"ContainerStarted","Data":"e54f5e7d0247f0815eeaef46d93b8f4810dac5238f46480f5381cd91b1ad8d73"} Jul 10 20:16:53.351978 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.351947 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-68ll2" event={"ID":"87a01125-6085-4468-bf2e-523adaa9055e","Type":"ContainerStarted","Data":"a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839"} Jul 10 20:16:53.352064 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.351992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-68ll2" event={"ID":"87a01125-6085-4468-bf2e-523adaa9055e","Type":"ContainerStarted","Data":"ba76fe34d27cf6c44c147efbbaf97b5207b6c6cbecbe34994881331560776c90"} Jul 10 20:16:53.352064 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.352042 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-68ll2" podUID="87a01125-6085-4468-bf2e-523adaa9055e" containerName="authorino" containerID="cri-o://a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839" gracePeriod=30 Jul 10 20:16:53.367542 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.367489 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-68ll2" podStartSLOduration=0.92505051 podStartE2EDuration="1.36747286s" totalImagesPullingTime="442.42235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:16:53.366617395 +0000 UTC m=+493.839586721" watchObservedRunningTime="2026-07-10 20:16:53.36747286 +0000 UTC m=+493.840442244" Jul 10 20:16:53.489586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.489497 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cbb87d59-lcrw8"] Jul 10 20:16:53.491963 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:16:53.491916 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13d21514_c497_4de4_a876_2621c35bc0d8.slice/crio-83ff3951c407fa254c556844d6d431c295de1918747d276255e722623d1f8ac1 WatchSource:0}: Error finding container 83ff3951c407fa254c556844d6d431c295de1918747d276255e722623d1f8ac1: Status 404 returned error can't find the container with id 83ff3951c407fa254c556844d6d431c295de1918747d276255e722623d1f8ac1 Jul 10 20:16:53.632140 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.632108 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-68ll2" Jul 10 20:16:53.689267 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.689179 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqxt9\" (UniqueName: \"kubernetes.io/projected/87a01125-6085-4468-bf2e-523adaa9055e-kube-api-access-xqxt9\") pod \"87a01125-6085-4468-bf2e-523adaa9055e\" (UID: \"87a01125-6085-4468-bf2e-523adaa9055e\") " Jul 10 20:16:53.691452 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.691419 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a01125-6085-4468-bf2e-523adaa9055e-kube-api-access-xqxt9" (OuterVolumeSpecName: "kube-api-access-xqxt9") pod "87a01125-6085-4468-bf2e-523adaa9055e" (UID: "87a01125-6085-4468-bf2e-523adaa9055e"). InnerVolumeSpecName "kube-api-access-xqxt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:16:53.790727 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:53.790692 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqxt9\" (UniqueName: \"kubernetes.io/projected/87a01125-6085-4468-bf2e-523adaa9055e-kube-api-access-xqxt9\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:16:54.361014 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.360909 2565 generic.go:356] "Generic (PLEG): container finished" podID="87a01125-6085-4468-bf2e-523adaa9055e" containerID="a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839" exitCode=0 Jul 10 20:16:54.361014 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.360956 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-68ll2" Jul 10 20:16:54.361014 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.360999 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-68ll2" event={"ID":"87a01125-6085-4468-bf2e-523adaa9055e","Type":"ContainerDied","Data":"a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839"} Jul 10 20:16:54.361540 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.361045 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-68ll2" event={"ID":"87a01125-6085-4468-bf2e-523adaa9055e","Type":"ContainerDied","Data":"ba76fe34d27cf6c44c147efbbaf97b5207b6c6cbecbe34994881331560776c90"} Jul 10 20:16:54.361540 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.361067 2565 scope.go:117] "RemoveContainer" containerID="a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839" Jul 10 20:16:54.362652 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.362624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" event={"ID":"13d21514-c497-4de4-a876-2621c35bc0d8","Type":"ContainerStarted","Data":"54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577"} Jul 10 20:16:54.362769 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.362667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" event={"ID":"13d21514-c497-4de4-a876-2621c35bc0d8","Type":"ContainerStarted","Data":"83ff3951c407fa254c556844d6d431c295de1918747d276255e722623d1f8ac1"} Jul 10 20:16:54.364787 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.364760 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" event={"ID":"ff974267-84bc-4c76-8da7-6196f8005911","Type":"ContainerStarted","Data":"3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a"} Jul 10 20:16:54.364956 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.364764 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" podUID="ff974267-84bc-4c76-8da7-6196f8005911" containerName="authorino" containerID="cri-o://3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a" gracePeriod=30 Jul 10 20:16:54.371075 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.371053 2565 scope.go:117] "RemoveContainer" containerID="a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839" Jul 10 20:16:54.371368 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:16:54.371349 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839\": container with ID starting with a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839 not found: ID does not exist" containerID="a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839" Jul 10 20:16:54.371421 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.371379 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839"} err="failed to get container status \"a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839\": rpc error: code = NotFound desc = could not find container \"a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839\": container with ID starting with a071a2c1dd725d7ebeed12619d789820aae3617d2dc14ec96a72dec98da65839 not found: ID does not exist" Jul 10 20:16:54.375108 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.375084 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-68ll2"] Jul 10 20:16:54.380124 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.380100 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-68ll2"] Jul 10 20:16:54.392434 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.392384 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" podStartSLOduration=0.967933689 podStartE2EDuration="1.392367104s" totalImagesPullingTime="424.433415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:16:54.39140174 +0000 UTC m=+494.864371066" watchObservedRunningTime="2026-07-10 20:16:54.392367104 +0000 UTC m=+494.865336429" Jul 10 20:16:54.405504 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.405441 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" podStartSLOduration=2.057117148 podStartE2EDuration="2.405425728s" totalImagesPullingTime="348.30858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:16:54.404804171 +0000 UTC m=+494.877773498" watchObservedRunningTime="2026-07-10 20:16:54.405425728 +0000 UTC m=+494.878395053" Jul 10 20:16:54.419947 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.419904 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8fp2n"] Jul 10 20:16:54.420136 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.420081 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-8fp2n" podUID="6edc5121-cce3-4a41-8be2-b8289fe14851" containerName="authorino" containerID="cri-o://57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f" gracePeriod=30 Jul 10 20:16:54.691784 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.691757 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" Jul 10 20:16:54.694850 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.694824 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8fp2n" Jul 10 20:16:54.798500 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.798466 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmsc7\" (UniqueName: \"kubernetes.io/projected/ff974267-84bc-4c76-8da7-6196f8005911-kube-api-access-lmsc7\") pod \"ff974267-84bc-4c76-8da7-6196f8005911\" (UID: \"ff974267-84bc-4c76-8da7-6196f8005911\") " Jul 10 20:16:54.798673 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.798512 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-247zh\" (UniqueName: \"kubernetes.io/projected/6edc5121-cce3-4a41-8be2-b8289fe14851-kube-api-access-247zh\") pod \"6edc5121-cce3-4a41-8be2-b8289fe14851\" (UID: \"6edc5121-cce3-4a41-8be2-b8289fe14851\") " Jul 10 20:16:54.800751 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.800716 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff974267-84bc-4c76-8da7-6196f8005911-kube-api-access-lmsc7" (OuterVolumeSpecName: "kube-api-access-lmsc7") pod "ff974267-84bc-4c76-8da7-6196f8005911" (UID: "ff974267-84bc-4c76-8da7-6196f8005911"). InnerVolumeSpecName "kube-api-access-lmsc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:16:54.800902 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.800790 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edc5121-cce3-4a41-8be2-b8289fe14851-kube-api-access-247zh" (OuterVolumeSpecName: "kube-api-access-247zh") pod "6edc5121-cce3-4a41-8be2-b8289fe14851" (UID: "6edc5121-cce3-4a41-8be2-b8289fe14851"). InnerVolumeSpecName "kube-api-access-247zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:16:54.899769 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.899736 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmsc7\" (UniqueName: \"kubernetes.io/projected/ff974267-84bc-4c76-8da7-6196f8005911-kube-api-access-lmsc7\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:16:54.899769 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:54.899767 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-247zh\" (UniqueName: \"kubernetes.io/projected/6edc5121-cce3-4a41-8be2-b8289fe14851-kube-api-access-247zh\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:16:55.372167 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.372074 2565 generic.go:356] "Generic (PLEG): container finished" podID="ff974267-84bc-4c76-8da7-6196f8005911" containerID="3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a" exitCode=0 Jul 10 20:16:55.372167 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.372126 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" Jul 10 20:16:55.372167 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.372150 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" event={"ID":"ff974267-84bc-4c76-8da7-6196f8005911","Type":"ContainerDied","Data":"3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a"} Jul 10 20:16:55.372655 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.372186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d7d9c75f-hs4p8" event={"ID":"ff974267-84bc-4c76-8da7-6196f8005911","Type":"ContainerDied","Data":"e54f5e7d0247f0815eeaef46d93b8f4810dac5238f46480f5381cd91b1ad8d73"} Jul 10 20:16:55.372655 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.372203 2565 scope.go:117] "RemoveContainer" containerID="3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a" Jul 10 20:16:55.373487 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.373462 2565 generic.go:356] "Generic (PLEG): container finished" podID="6edc5121-cce3-4a41-8be2-b8289fe14851" containerID="57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f" exitCode=0 Jul 10 20:16:55.373606 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.373512 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8fp2n" Jul 10 20:16:55.373606 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.373541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8fp2n" event={"ID":"6edc5121-cce3-4a41-8be2-b8289fe14851","Type":"ContainerDied","Data":"57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f"} Jul 10 20:16:55.373606 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.373576 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8fp2n" event={"ID":"6edc5121-cce3-4a41-8be2-b8289fe14851","Type":"ContainerDied","Data":"a0df1eb1e34012963efdf68b8cac8f817ad4d89e9e2fa531fdb6df7745d1fc48"} Jul 10 20:16:55.381208 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.381185 2565 scope.go:117] "RemoveContainer" containerID="3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a" Jul 10 20:16:55.381607 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:16:55.381449 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a\": container with ID starting with 3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a not found: ID does not exist" containerID="3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a" Jul 10 20:16:55.381607 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.381475 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a"} err="failed to get container status \"3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a\": rpc error: code = NotFound desc = could not find container \"3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a\": container with ID starting with 3151daa140c879a54ecc8ae4e9730d8d59753a306ca5f504c835aea7ad23785a not found: ID does not exist" Jul 10 20:16:55.381607 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.381491 2565 scope.go:117] "RemoveContainer" containerID="57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f" Jul 10 20:16:55.389467 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.389444 2565 scope.go:117] "RemoveContainer" containerID="57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f" Jul 10 20:16:55.389734 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:16:55.389714 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f\": container with ID starting with 57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f not found: ID does not exist" containerID="57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f" Jul 10 20:16:55.389802 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.389744 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f"} err="failed to get container status \"57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f\": rpc error: code = NotFound desc = could not find container \"57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f\": container with ID starting with 57eb08efad7d5f396f7827b87a2146e58f29468c4619038c279eb2a2e361370f not found: ID does not exist" Jul 10 20:16:55.395196 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.395169 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8fp2n"] Jul 10 20:16:55.398706 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.398683 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8fp2n"] Jul 10 20:16:55.409928 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.409890 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74d7d9c75f-hs4p8"] Jul 10 20:16:55.413903 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:55.413855 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74d7d9c75f-hs4p8"] Jul 10 20:16:56.040264 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:56.040231 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6edc5121-cce3-4a41-8be2-b8289fe14851" path="/var/lib/kubelet/pods/6edc5121-cce3-4a41-8be2-b8289fe14851/volumes" Jul 10 20:16:56.040579 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:56.040566 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a01125-6085-4468-bf2e-523adaa9055e" path="/var/lib/kubelet/pods/87a01125-6085-4468-bf2e-523adaa9055e/volumes" Jul 10 20:16:56.040849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:16:56.040837 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff974267-84bc-4c76-8da7-6196f8005911" path="/var/lib/kubelet/pods/ff974267-84bc-4c76-8da7-6196f8005911/volumes" Jul 10 20:17:57.430452 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430419 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4"] Jul 10 20:17:57.430919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430751 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff974267-84bc-4c76-8da7-6196f8005911" containerName="authorino" Jul 10 20:17:57.430919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430762 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff974267-84bc-4c76-8da7-6196f8005911" containerName="authorino" Jul 10 20:17:57.430919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430772 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edc5121-cce3-4a41-8be2-b8289fe14851" containerName="authorino" Jul 10 20:17:57.430919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430777 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edc5121-cce3-4a41-8be2-b8289fe14851" containerName="authorino" Jul 10 20:17:57.430919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430797 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a01125-6085-4468-bf2e-523adaa9055e" containerName="authorino" Jul 10 20:17:57.430919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430802 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a01125-6085-4468-bf2e-523adaa9055e" containerName="authorino" Jul 10 20:17:57.430919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430880 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edc5121-cce3-4a41-8be2-b8289fe14851" containerName="authorino" Jul 10 20:17:57.430919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430891 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a01125-6085-4468-bf2e-523adaa9055e" containerName="authorino" Jul 10 20:17:57.430919 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.430901 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff974267-84bc-4c76-8da7-6196f8005911" containerName="authorino" Jul 10 20:17:57.435284 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.435259 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.438380 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.438352 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 20:17:57.438524 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.438354 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 20:17:57.438524 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.438360 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wfnt5\"" Jul 10 20:17:57.438524 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.438390 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 20:17:57.441817 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.441792 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4"] Jul 10 20:17:57.562140 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.562097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.562305 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.562180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.562305 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.562202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.562305 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.562223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9xv\" (UniqueName: \"kubernetes.io/projected/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-kube-api-access-jf9xv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.562305 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.562246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.562305 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.562260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.662720 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.662690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.662924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.662756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.662924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.662783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.662924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.662802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9xv\" (UniqueName: \"kubernetes.io/projected/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-kube-api-access-jf9xv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.662924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.662827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.662924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.662851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.663193 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.663171 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.663265 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.663198 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.663265 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.663217 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.665359 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.665327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.665495 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.665479 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.670586 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.670566 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9xv\" (UniqueName: \"kubernetes.io/projected/6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8-kube-api-access-jf9xv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dzlb4\" (UID: \"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.747402 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.747308 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:17:57.890553 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:57.890529 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4"] Jul 10 20:17:57.893217 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:17:57.893183 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0af3ed_e5c7_4bbf_b046_f37b9e0a72f8.slice/crio-bc0edd76ce020233b9a0430085b8efda6719ca78f4313921a9e40322841964f3 WatchSource:0}: Error finding container bc0edd76ce020233b9a0430085b8efda6719ca78f4313921a9e40322841964f3: Status 404 returned error can't find the container with id bc0edd76ce020233b9a0430085b8efda6719ca78f4313921a9e40322841964f3 Jul 10 20:17:58.669752 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:58.669697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" event={"ID":"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8","Type":"ContainerStarted","Data":"bc0edd76ce020233b9a0430085b8efda6719ca78f4313921a9e40322841964f3"} Jul 10 20:17:59.675464 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:59.675425 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" event={"ID":"6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8","Type":"ContainerStarted","Data":"a6ea33977bd2b227faab2aa1f317843a3d4b55d8045299ec2700838fffdde2ca"} Jul 10 20:17:59.692588 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:17:59.692533 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" podStartSLOduration=1.533627573 podStartE2EDuration="2.692515061s" totalImagesPullingTime="1.158887488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:17:59.691301438 +0000 UTC m=+560.164270768" watchObservedRunningTime="2026-07-10 20:17:59.692515061 +0000 UTC m=+560.165484388" Jul 10 20:18:07.747615 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:07.747577 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:18:07.748085 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:07.747641 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:18:07.760294 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:07.760263 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:18:08.733426 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:08.733394 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dzlb4" Jul 10 20:18:35.019401 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.019366 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf"] Jul 10 20:18:35.022223 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.022194 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.024621 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.024598 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 20:18:35.032113 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.032087 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf"] Jul 10 20:18:35.097246 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.097195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.097432 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.097284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.097432 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.097311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.097432 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.097358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.097432 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.097406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3945e298-4d86-4c35-bf50-bee3416b6c0e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.097569 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.097433 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vsxd\" (UniqueName: \"kubernetes.io/projected/3945e298-4d86-4c35-bf50-bee3416b6c0e-kube-api-access-2vsxd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.197961 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.197930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.198113 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.197970 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.198113 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.197992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.198113 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.198018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3945e298-4d86-4c35-bf50-bee3416b6c0e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.198113 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.198046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsxd\" (UniqueName: \"kubernetes.io/projected/3945e298-4d86-4c35-bf50-bee3416b6c0e-kube-api-access-2vsxd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.198310 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.198133 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.198488 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.198461 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.198624 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.198492 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.198624 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.198514 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.200501 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.200474 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3945e298-4d86-4c35-bf50-bee3416b6c0e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.200722 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.200704 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3945e298-4d86-4c35-bf50-bee3416b6c0e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.206594 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.206565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vsxd\" (UniqueName: \"kubernetes.io/projected/3945e298-4d86-4c35-bf50-bee3416b6c0e-kube-api-access-2vsxd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf\" (UID: \"3945e298-4d86-4c35-bf50-bee3416b6c0e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.334013 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.333920 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:35.474116 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.474090 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf"] Jul 10 20:18:35.477675 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:18:35.477640 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3945e298_4d86_4c35_bf50_bee3416b6c0e.slice/crio-dc507849f48dd9883ea05a451a52ff67f6d219c21fad8ebbca56a0514e0ec702 WatchSource:0}: Error finding container dc507849f48dd9883ea05a451a52ff67f6d219c21fad8ebbca56a0514e0ec702: Status 404 returned error can't find the container with id dc507849f48dd9883ea05a451a52ff67f6d219c21fad8ebbca56a0514e0ec702 Jul 10 20:18:35.846141 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.846103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" event={"ID":"3945e298-4d86-4c35-bf50-bee3416b6c0e","Type":"ContainerStarted","Data":"a157bcdb60fd31aeed1ea65bcde41e8cedfc1949a7e722924593a658d62d211c"} Jul 10 20:18:35.846311 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.846160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" event={"ID":"3945e298-4d86-4c35-bf50-bee3416b6c0e","Type":"ContainerStarted","Data":"dc507849f48dd9883ea05a451a52ff67f6d219c21fad8ebbca56a0514e0ec702"} Jul 10 20:18:35.863140 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:35.863084 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" podStartSLOduration=1.6989717899999999 podStartE2EDuration="1.86306763s" totalImagesPullingTime="164.09584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:18:35.862065157 +0000 UTC m=+596.335034483" watchObservedRunningTime="2026-07-10 20:18:35.86306763 +0000 UTC m=+596.336036955" Jul 10 20:18:39.923255 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:39.923226 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:18:39.924776 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:39.924743 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:18:45.334138 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:45.334083 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:45.336747 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:45.334164 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:45.346998 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:45.346973 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:45.902538 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:45.902505 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf" Jul 10 20:18:52.731411 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.731370 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth"] Jul 10 20:18:52.734709 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.734688 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.737150 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.737126 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 20:18:52.745490 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.745458 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth"] Jul 10 20:18:52.863165 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.863124 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.863356 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.863190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.863356 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.863224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.863356 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.863258 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l57j4\" (UniqueName: \"kubernetes.io/projected/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-kube-api-access-l57j4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.863467 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.863345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.863467 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.863386 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.964796 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.964755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57j4\" (UniqueName: \"kubernetes.io/projected/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-kube-api-access-l57j4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.965011 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.964808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.965011 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.964827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.965011 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.964896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.965011 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.964938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.965011 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.964959 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.965361 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.965337 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.965361 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.965350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.965461 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.965416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.967450 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.967422 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.967651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.967634 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:52.973022 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:52.972987 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57j4\" (UniqueName: \"kubernetes.io/projected/a1fd351d-4151-4bdf-b2b5-f71bf71e123c-kube-api-access-l57j4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jbnth\" (UID: \"a1fd351d-4151-4bdf-b2b5-f71bf71e123c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:53.046764 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:53.046673 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:18:53.187615 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:53.187590 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth"] Jul 10 20:18:53.189762 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:18:53.189726 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fd351d_4151_4bdf_b2b5_f71bf71e123c.slice/crio-6b9962106da4d9cfda4614cbb258256260b7611dc884c380d45a08d76b34918f WatchSource:0}: Error finding container 6b9962106da4d9cfda4614cbb258256260b7611dc884c380d45a08d76b34918f: Status 404 returned error can't find the container with id 6b9962106da4d9cfda4614cbb258256260b7611dc884c380d45a08d76b34918f Jul 10 20:18:53.930259 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:53.930223 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" event={"ID":"a1fd351d-4151-4bdf-b2b5-f71bf71e123c","Type":"ContainerStarted","Data":"165c6d4f9eb38e1b9d02eb0530a46b9f29ddcb0df7f9281fe678590fbf924dfb"} Jul 10 20:18:53.930681 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:53.930310 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" event={"ID":"a1fd351d-4151-4bdf-b2b5-f71bf71e123c","Type":"ContainerStarted","Data":"6b9962106da4d9cfda4614cbb258256260b7611dc884c380d45a08d76b34918f"} Jul 10 20:18:53.946800 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:53.946747 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" podStartSLOduration=1.774192666 podStartE2EDuration="1.946728198s" totalImagesPullingTime="172.535532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:18:53.945385573 +0000 UTC m=+614.418354900" watchObservedRunningTime="2026-07-10 20:18:53.946728198 +0000 UTC m=+614.419697523" Jul 10 20:18:58.931450 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:58.931419 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8"] Jul 10 20:18:58.934973 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:58.934950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:58.937681 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:58.937657 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 20:18:58.944877 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:58.944825 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8"] Jul 10 20:18:59.015442 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.015398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.015618 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.015459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.015618 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.015514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.015618 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.015576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.015618 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.015595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdbl\" (UniqueName: \"kubernetes.io/projected/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-kube-api-access-dkdbl\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.015768 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.015638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.116234 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.116194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.116434 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.116280 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.116434 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.116327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.116434 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.116354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.116434 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.116427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.116638 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.116451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdbl\" (UniqueName: \"kubernetes.io/projected/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-kube-api-access-dkdbl\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.116638 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.116614 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.116799 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.116779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.116933 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.116911 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.118876 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.118832 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.119083 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.119066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.124112 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.124086 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdbl\" (UniqueName: \"kubernetes.io/projected/854748ac-51f1-401f-9b22-eb2ba1b2bc9f-kube-api-access-dkdbl\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8\" (UID: \"854748ac-51f1-401f-9b22-eb2ba1b2bc9f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.246425 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.246322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:18:59.389485 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.389406 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8"] Jul 10 20:18:59.391881 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:18:59.391833 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod854748ac_51f1_401f_9b22_eb2ba1b2bc9f.slice/crio-b712f7af806728f8622dd241c43502c120206d58b095009c1ee24f96a14928c5 WatchSource:0}: Error finding container b712f7af806728f8622dd241c43502c120206d58b095009c1ee24f96a14928c5: Status 404 returned error can't find the container with id b712f7af806728f8622dd241c43502c120206d58b095009c1ee24f96a14928c5 Jul 10 20:18:59.961342 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.961301 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" event={"ID":"854748ac-51f1-401f-9b22-eb2ba1b2bc9f","Type":"ContainerStarted","Data":"7c0735cb191b37f368e8dd2d9310fcff82412f8a9ac0af3940d59a910344e2b8"} Jul 10 20:18:59.961740 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.961363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" event={"ID":"854748ac-51f1-401f-9b22-eb2ba1b2bc9f","Type":"ContainerStarted","Data":"b712f7af806728f8622dd241c43502c120206d58b095009c1ee24f96a14928c5"} Jul 10 20:18:59.978935 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:18:59.978882 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" podStartSLOduration=1.807785463 podStartE2EDuration="1.978850098s" totalImagesPullingTime="171.064635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:18:59.977722027 +0000 UTC m=+620.450691354" watchObservedRunningTime="2026-07-10 20:18:59.978850098 +0000 UTC m=+620.451819423" Jul 10 20:19:03.047490 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:03.047452 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:19:03.047940 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:03.047519 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:19:03.063261 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:03.063227 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:19:03.996173 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:03.996141 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jbnth" Jul 10 20:19:09.246475 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:09.246433 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:19:09.246932 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:09.246517 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:19:09.259265 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:09.259234 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:19:10.025021 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:10.024991 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8" Jul 10 20:19:34.249071 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.249033 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75f695b566-2f42g"] Jul 10 20:19:34.252443 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.252423 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75f695b566-2f42g" Jul 10 20:19:34.262613 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.262587 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75f695b566-2f42g"] Jul 10 20:19:34.361041 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.361001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkw4c\" (UniqueName: \"kubernetes.io/projected/2ff8108c-9747-4533-a7d9-2378bcfcb9c9-kube-api-access-wkw4c\") pod \"authorino-75f695b566-2f42g\" (UID: \"2ff8108c-9747-4533-a7d9-2378bcfcb9c9\") " pod="kuadrant-system/authorino-75f695b566-2f42g" Jul 10 20:19:34.361256 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.361067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2ff8108c-9747-4533-a7d9-2378bcfcb9c9-tls-cert\") pod \"authorino-75f695b566-2f42g\" (UID: \"2ff8108c-9747-4533-a7d9-2378bcfcb9c9\") " pod="kuadrant-system/authorino-75f695b566-2f42g" Jul 10 20:19:34.461888 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.461829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkw4c\" (UniqueName: \"kubernetes.io/projected/2ff8108c-9747-4533-a7d9-2378bcfcb9c9-kube-api-access-wkw4c\") pod \"authorino-75f695b566-2f42g\" (UID: \"2ff8108c-9747-4533-a7d9-2378bcfcb9c9\") " pod="kuadrant-system/authorino-75f695b566-2f42g" Jul 10 20:19:34.462100 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.461937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2ff8108c-9747-4533-a7d9-2378bcfcb9c9-tls-cert\") pod \"authorino-75f695b566-2f42g\" (UID: \"2ff8108c-9747-4533-a7d9-2378bcfcb9c9\") " pod="kuadrant-system/authorino-75f695b566-2f42g" Jul 10 20:19:34.464750 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.464723 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2ff8108c-9747-4533-a7d9-2378bcfcb9c9-tls-cert\") pod \"authorino-75f695b566-2f42g\" (UID: \"2ff8108c-9747-4533-a7d9-2378bcfcb9c9\") " pod="kuadrant-system/authorino-75f695b566-2f42g" Jul 10 20:19:34.469458 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.469419 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkw4c\" (UniqueName: \"kubernetes.io/projected/2ff8108c-9747-4533-a7d9-2378bcfcb9c9-kube-api-access-wkw4c\") pod \"authorino-75f695b566-2f42g\" (UID: \"2ff8108c-9747-4533-a7d9-2378bcfcb9c9\") " pod="kuadrant-system/authorino-75f695b566-2f42g" Jul 10 20:19:34.563923 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.563834 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75f695b566-2f42g" Jul 10 20:19:34.700544 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:34.700518 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75f695b566-2f42g"] Jul 10 20:19:34.703278 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:19:34.703244 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ff8108c_9747_4533_a7d9_2378bcfcb9c9.slice/crio-f5db4c05fe40bda8e26b78121750c179546e20f52d06a93cda6336c33a5d61c4 WatchSource:0}: Error finding container f5db4c05fe40bda8e26b78121750c179546e20f52d06a93cda6336c33a5d61c4: Status 404 returned error can't find the container with id f5db4c05fe40bda8e26b78121750c179546e20f52d06a93cda6336c33a5d61c4 Jul 10 20:19:35.133140 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:35.133101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75f695b566-2f42g" event={"ID":"2ff8108c-9747-4533-a7d9-2378bcfcb9c9","Type":"ContainerStarted","Data":"f5db4c05fe40bda8e26b78121750c179546e20f52d06a93cda6336c33a5d61c4"} Jul 10 20:19:36.141747 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.141706 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75f695b566-2f42g" event={"ID":"2ff8108c-9747-4533-a7d9-2378bcfcb9c9","Type":"ContainerStarted","Data":"c20b7a9366af243a93dda96ea158743fbacbd14c1807b3794fc0543d0461b48c"} Jul 10 20:19:36.158079 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.158011 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75f695b566-2f42g" podStartSLOduration=1.673663428 podStartE2EDuration="2.157992035s" totalImagesPullingTime="484.328607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:19:36.156668801 +0000 UTC m=+656.629638131" watchObservedRunningTime="2026-07-10 20:19:36.157992035 +0000 UTC m=+656.630961360" Jul 10 20:19:36.190368 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.190295 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cbb87d59-lcrw8"] Jul 10 20:19:36.190560 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.190479 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" podUID="13d21514-c497-4de4-a876-2621c35bc0d8" containerName="authorino" containerID="cri-o://54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577" gracePeriod=30 Jul 10 20:19:36.459764 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.459737 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:19:36.483682 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.483649 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13d21514-c497-4de4-a876-2621c35bc0d8-tls-cert\") pod \"13d21514-c497-4de4-a876-2621c35bc0d8\" (UID: \"13d21514-c497-4de4-a876-2621c35bc0d8\") " Jul 10 20:19:36.483844 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.483691 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt6ql\" (UniqueName: \"kubernetes.io/projected/13d21514-c497-4de4-a876-2621c35bc0d8-kube-api-access-vt6ql\") pod \"13d21514-c497-4de4-a876-2621c35bc0d8\" (UID: \"13d21514-c497-4de4-a876-2621c35bc0d8\") " Jul 10 20:19:36.486236 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.486199 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d21514-c497-4de4-a876-2621c35bc0d8-kube-api-access-vt6ql" (OuterVolumeSpecName: "kube-api-access-vt6ql") pod "13d21514-c497-4de4-a876-2621c35bc0d8" (UID: "13d21514-c497-4de4-a876-2621c35bc0d8"). InnerVolumeSpecName "kube-api-access-vt6ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:19:36.495962 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.495921 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d21514-c497-4de4-a876-2621c35bc0d8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "13d21514-c497-4de4-a876-2621c35bc0d8" (UID: "13d21514-c497-4de4-a876-2621c35bc0d8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:19:36.584705 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.584663 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/13d21514-c497-4de4-a876-2621c35bc0d8-tls-cert\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:19:36.584705 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:36.584708 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt6ql\" (UniqueName: \"kubernetes.io/projected/13d21514-c497-4de4-a876-2621c35bc0d8-kube-api-access-vt6ql\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:19:37.149010 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:37.148969 2565 generic.go:356] "Generic (PLEG): container finished" podID="13d21514-c497-4de4-a876-2621c35bc0d8" containerID="54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577" exitCode=0 Jul 10 20:19:37.149456 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:37.149017 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" Jul 10 20:19:37.149456 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:37.149057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" event={"ID":"13d21514-c497-4de4-a876-2621c35bc0d8","Type":"ContainerDied","Data":"54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577"} Jul 10 20:19:37.149456 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:37.149103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cbb87d59-lcrw8" event={"ID":"13d21514-c497-4de4-a876-2621c35bc0d8","Type":"ContainerDied","Data":"83ff3951c407fa254c556844d6d431c295de1918747d276255e722623d1f8ac1"} Jul 10 20:19:37.149456 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:37.149122 2565 scope.go:117] "RemoveContainer" containerID="54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577" Jul 10 20:19:37.158597 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:37.158572 2565 scope.go:117] "RemoveContainer" containerID="54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577" Jul 10 20:19:37.159285 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:19:37.158994 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577\": container with ID starting with 54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577 not found: ID does not exist" containerID="54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577" Jul 10 20:19:37.159285 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:37.159031 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577"} err="failed to get container status \"54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577\": rpc error: code = NotFound desc = could not find container \"54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577\": container with ID starting with 54972a046fbf940e61f398e6fa5f171864ede103c0263a604b34018d2369f577 not found: ID does not exist" Jul 10 20:19:37.176517 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:37.176475 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cbb87d59-lcrw8"] Jul 10 20:19:37.180072 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:37.180038 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cbb87d59-lcrw8"] Jul 10 20:19:38.040994 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:19:38.040955 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d21514-c497-4de4-a876-2621c35bc0d8" path="/var/lib/kubelet/pods/13d21514-c497-4de4-a876-2621c35bc0d8/volumes" Jul 10 20:23:39.951846 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:23:39.951774 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:23:39.954014 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:23:39.953992 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:25:33.082932 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:33.082895 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p"] Jul 10 20:25:33.084974 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:33.083099 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" podUID="98842af1-aacd-4334-a5e5-b26e486a86ee" containerName="manager" containerID="cri-o://db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e" gracePeriod=10 Jul 10 20:25:36.446421 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.446389 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:25:36.498412 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.498374 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whppc\" (UniqueName: \"kubernetes.io/projected/98842af1-aacd-4334-a5e5-b26e486a86ee-kube-api-access-whppc\") pod \"98842af1-aacd-4334-a5e5-b26e486a86ee\" (UID: \"98842af1-aacd-4334-a5e5-b26e486a86ee\") " Jul 10 20:25:36.498609 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.498462 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98842af1-aacd-4334-a5e5-b26e486a86ee-extensions-socket-volume\") pod \"98842af1-aacd-4334-a5e5-b26e486a86ee\" (UID: \"98842af1-aacd-4334-a5e5-b26e486a86ee\") " Jul 10 20:25:36.498839 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.498807 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98842af1-aacd-4334-a5e5-b26e486a86ee-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "98842af1-aacd-4334-a5e5-b26e486a86ee" (UID: "98842af1-aacd-4334-a5e5-b26e486a86ee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:25:36.500832 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.500801 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98842af1-aacd-4334-a5e5-b26e486a86ee-kube-api-access-whppc" (OuterVolumeSpecName: "kube-api-access-whppc") pod "98842af1-aacd-4334-a5e5-b26e486a86ee" (UID: "98842af1-aacd-4334-a5e5-b26e486a86ee"). InnerVolumeSpecName "kube-api-access-whppc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:25:36.600109 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.600014 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whppc\" (UniqueName: \"kubernetes.io/projected/98842af1-aacd-4334-a5e5-b26e486a86ee-kube-api-access-whppc\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:25:36.600109 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.600044 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98842af1-aacd-4334-a5e5-b26e486a86ee-extensions-socket-volume\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:25:36.926911 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.926852 2565 generic.go:356] "Generic (PLEG): container finished" podID="98842af1-aacd-4334-a5e5-b26e486a86ee" containerID="db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e" exitCode=0 Jul 10 20:25:36.927087 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.926925 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" Jul 10 20:25:36.927087 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.926936 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" event={"ID":"98842af1-aacd-4334-a5e5-b26e486a86ee","Type":"ContainerDied","Data":"db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e"} Jul 10 20:25:36.927087 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.926990 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p" event={"ID":"98842af1-aacd-4334-a5e5-b26e486a86ee","Type":"ContainerDied","Data":"9297da4e8f80a374b3e2b62211453ed38e91af6eee82053ed5017f278dcb3595"} Jul 10 20:25:36.927087 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.927012 2565 scope.go:117] "RemoveContainer" containerID="db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e" Jul 10 20:25:36.936893 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.936841 2565 scope.go:117] "RemoveContainer" containerID="db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e" Jul 10 20:25:36.937263 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:25:36.937239 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e\": container with ID starting with db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e not found: ID does not exist" containerID="db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e" Jul 10 20:25:36.937345 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.937276 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e"} err="failed to get container status \"db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e\": rpc error: code = NotFound desc = could not find container \"db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e\": container with ID starting with db82cdcd76a5f47e5eb95a1ddfe11c66b05bc9b09d8ca5031dae6c4b65968f6e not found: ID does not exist" Jul 10 20:25:36.948999 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.948964 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p"] Jul 10 20:25:36.954165 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:36.954135 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g6v6p"] Jul 10 20:25:38.040642 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:25:38.040597 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98842af1-aacd-4334-a5e5-b26e486a86ee" path="/var/lib/kubelet/pods/98842af1-aacd-4334-a5e5-b26e486a86ee/volumes" Jul 10 20:26:34.244582 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.244494 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc"] Jul 10 20:26:34.245096 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.244844 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98842af1-aacd-4334-a5e5-b26e486a86ee" containerName="manager" Jul 10 20:26:34.245096 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.244855 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="98842af1-aacd-4334-a5e5-b26e486a86ee" containerName="manager" Jul 10 20:26:34.245096 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.244880 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d21514-c497-4de4-a876-2621c35bc0d8" containerName="authorino" Jul 10 20:26:34.245096 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.244889 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d21514-c497-4de4-a876-2621c35bc0d8" containerName="authorino" Jul 10 20:26:34.245096 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.244969 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d21514-c497-4de4-a876-2621c35bc0d8" containerName="authorino" Jul 10 20:26:34.245096 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.244979 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="98842af1-aacd-4334-a5e5-b26e486a86ee" containerName="manager" Jul 10 20:26:34.247498 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.247481 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:26:34.250168 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.250147 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z247m\"" Jul 10 20:26:34.262009 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.261977 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc"] Jul 10 20:26:34.333910 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.333830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b8rvc\" (UID: \"cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:26:34.334084 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.333995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6j5r\" (UniqueName: \"kubernetes.io/projected/cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d-kube-api-access-f6j5r\") pod \"kuadrant-operator-controller-manager-86878c769f-b8rvc\" (UID: \"cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:26:34.435038 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.434998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b8rvc\" (UID: \"cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:26:34.435200 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.435107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6j5r\" (UniqueName: \"kubernetes.io/projected/cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d-kube-api-access-f6j5r\") pod \"kuadrant-operator-controller-manager-86878c769f-b8rvc\" (UID: \"cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:26:34.435436 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.435415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b8rvc\" (UID: \"cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:26:34.445162 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.445131 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6j5r\" (UniqueName: \"kubernetes.io/projected/cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d-kube-api-access-f6j5r\") pod \"kuadrant-operator-controller-manager-86878c769f-b8rvc\" (UID: \"cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:26:34.558095 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.558003 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:26:34.701482 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:34.701456 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc"] Jul 10 20:26:34.704011 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:26:34.703984 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1a48af_a1ed_492f_a7a2_b246dcb4bd3d.slice/crio-98c0763360de6118db1baef0a3fb440f5f4e9c3eefa5e7977352fd235ed35831 WatchSource:0}: Error finding container 98c0763360de6118db1baef0a3fb440f5f4e9c3eefa5e7977352fd235ed35831: Status 404 returned error can't find the container with id 98c0763360de6118db1baef0a3fb440f5f4e9c3eefa5e7977352fd235ed35831 Jul 10 20:26:35.208745 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:35.208702 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" event={"ID":"cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d","Type":"ContainerStarted","Data":"596e0cd84ee6094fa1b0d277427624fabd62f97f080050a4cc9db9a85cb12c4b"} Jul 10 20:26:35.208991 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:35.208852 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:26:35.208991 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:35.208921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" event={"ID":"cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d","Type":"ContainerStarted","Data":"98c0763360de6118db1baef0a3fb440f5f4e9c3eefa5e7977352fd235ed35831"} Jul 10 20:26:35.242054 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:35.241999 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" podStartSLOduration=1.241981335 podStartE2EDuration="1.241981335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:26:35.241131622 +0000 UTC m=+1075.714100959" watchObservedRunningTime="2026-07-10 20:26:35.241981335 +0000 UTC m=+1075.714950661" Jul 10 20:26:46.216392 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:26:46.216359 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8rvc" Jul 10 20:28:39.981114 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:28:39.981071 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:28:39.984042 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:28:39.984019 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:33:40.012790 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:33:40.012760 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:33:40.016647 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:33:40.016622 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:36:09.757783 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.757705 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67"] Jul 10 20:36:09.761221 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.761198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.763893 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.763841 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-02c11db1\"/\"openshift-service-ca.crt\"" Jul 10 20:36:09.764041 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.763901 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-02c11db1\"/\"e2e-lifecycle-model-02c11db1-kserve-self-signed-certs\"" Jul 10 20:36:09.764531 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.764513 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-02c11db1\"/\"kube-root-ca.crt\"" Jul 10 20:36:09.764818 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.764799 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-02c11db1\"/\"default-dockercfg-778nk\"" Jul 10 20:36:09.769639 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.769425 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67"] Jul 10 20:36:09.875277 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.875236 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-tmp-dir\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.875452 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.875295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ts5l\" (UniqueName: \"kubernetes.io/projected/4daadd53-1e04-4b2e-a073-b7dda180f82f-kube-api-access-4ts5l\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.875452 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.875336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4daadd53-1e04-4b2e-a073-b7dda180f82f-tls-certs\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.875452 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.875368 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-home\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.875452 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.875385 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-model-cache\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.875583 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.875466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-dshm\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.975994 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.975952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4daadd53-1e04-4b2e-a073-b7dda180f82f-tls-certs\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.976178 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.976006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-home\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.976178 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.976027 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-model-cache\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.976178 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.976058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-dshm\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.976178 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.976103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-tmp-dir\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.976178 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.976155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ts5l\" (UniqueName: \"kubernetes.io/projected/4daadd53-1e04-4b2e-a073-b7dda180f82f-kube-api-access-4ts5l\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.976462 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.976437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-home\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.976531 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.976449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-model-cache\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.976590 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.976547 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-tmp-dir\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.978538 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.978507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-dshm\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.978849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.978829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4daadd53-1e04-4b2e-a073-b7dda180f82f-tls-certs\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:09.984152 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:09.984123 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ts5l\" (UniqueName: \"kubernetes.io/projected/4daadd53-1e04-4b2e-a073-b7dda180f82f-kube-api-access-4ts5l\") pod \"e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:10.073143 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:10.073049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:10.220441 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:10.220396 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67"] Jul 10 20:36:10.223676 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:36:10.223633 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4daadd53_1e04_4b2e_a073_b7dda180f82f.slice/crio-6a07ceafe7ff400b6087f24de0a0594ee9d2e85a21db86c844aa6c19bda45048 WatchSource:0}: Error finding container 6a07ceafe7ff400b6087f24de0a0594ee9d2e85a21db86c844aa6c19bda45048: Status 404 returned error can't find the container with id 6a07ceafe7ff400b6087f24de0a0594ee9d2e85a21db86c844aa6c19bda45048 Jul 10 20:36:11.020243 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:11.020199 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" event={"ID":"4daadd53-1e04-4b2e-a073-b7dda180f82f","Type":"ContainerStarted","Data":"05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a"} Jul 10 20:36:11.020243 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:11.020248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" event={"ID":"4daadd53-1e04-4b2e-a073-b7dda180f82f","Type":"ContainerStarted","Data":"6a07ceafe7ff400b6087f24de0a0594ee9d2e85a21db86c844aa6c19bda45048"} Jul 10 20:36:11.040287 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:11.040232 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" podStartSLOduration=2.04021361 podStartE2EDuration="2.04021361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:36:11.038593765 +0000 UTC m=+1651.511563093" watchObservedRunningTime="2026-07-10 20:36:11.04021361 +0000 UTC m=+1651.513182935" Jul 10 20:36:20.073995 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:20.073953 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:20.074394 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:20.074010 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:20.087008 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:20.086979 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:21.083147 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:21.083120 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:42.655155 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:42.655118 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67"] Jul 10 20:36:57.677657 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:57.677610 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" podUID="4daadd53-1e04-4b2e-a073-b7dda180f82f" containerName="main" containerID="cri-o://05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a" gracePeriod=45 Jul 10 20:36:57.938333 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:57.938267 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:58.020783 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.020743 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-model-cache\") pod \"4daadd53-1e04-4b2e-a073-b7dda180f82f\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " Jul 10 20:36:58.021019 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.020831 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-home\") pod \"4daadd53-1e04-4b2e-a073-b7dda180f82f\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " Jul 10 20:36:58.021019 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.020882 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4daadd53-1e04-4b2e-a073-b7dda180f82f-tls-certs\") pod \"4daadd53-1e04-4b2e-a073-b7dda180f82f\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " Jul 10 20:36:58.021019 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.020907 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-dshm\") pod \"4daadd53-1e04-4b2e-a073-b7dda180f82f\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " Jul 10 20:36:58.021019 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.020937 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ts5l\" (UniqueName: \"kubernetes.io/projected/4daadd53-1e04-4b2e-a073-b7dda180f82f-kube-api-access-4ts5l\") pod \"4daadd53-1e04-4b2e-a073-b7dda180f82f\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " Jul 10 20:36:58.021019 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.020980 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-tmp-dir\") pod \"4daadd53-1e04-4b2e-a073-b7dda180f82f\" (UID: \"4daadd53-1e04-4b2e-a073-b7dda180f82f\") " Jul 10 20:36:58.021251 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.021095 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-model-cache" (OuterVolumeSpecName: "model-cache") pod "4daadd53-1e04-4b2e-a073-b7dda180f82f" (UID: "4daadd53-1e04-4b2e-a073-b7dda180f82f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:36:58.021251 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.021114 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-home" (OuterVolumeSpecName: "home") pod "4daadd53-1e04-4b2e-a073-b7dda180f82f" (UID: "4daadd53-1e04-4b2e-a073-b7dda180f82f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:36:58.021341 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.021272 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4daadd53-1e04-4b2e-a073-b7dda180f82f" (UID: "4daadd53-1e04-4b2e-a073-b7dda180f82f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:36:58.021402 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.021382 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-home\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:36:58.021402 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.021396 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-tmp-dir\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:36:58.021492 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.021407 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-model-cache\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:36:58.023361 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.023335 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-dshm" (OuterVolumeSpecName: "dshm") pod "4daadd53-1e04-4b2e-a073-b7dda180f82f" (UID: "4daadd53-1e04-4b2e-a073-b7dda180f82f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:36:58.023361 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.023348 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4daadd53-1e04-4b2e-a073-b7dda180f82f-kube-api-access-4ts5l" (OuterVolumeSpecName: "kube-api-access-4ts5l") pod "4daadd53-1e04-4b2e-a073-b7dda180f82f" (UID: "4daadd53-1e04-4b2e-a073-b7dda180f82f"). InnerVolumeSpecName "kube-api-access-4ts5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:36:58.023492 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.023366 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4daadd53-1e04-4b2e-a073-b7dda180f82f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4daadd53-1e04-4b2e-a073-b7dda180f82f" (UID: "4daadd53-1e04-4b2e-a073-b7dda180f82f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:36:58.122363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.122321 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ts5l\" (UniqueName: \"kubernetes.io/projected/4daadd53-1e04-4b2e-a073-b7dda180f82f-kube-api-access-4ts5l\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:36:58.122363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.122362 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4daadd53-1e04-4b2e-a073-b7dda180f82f-tls-certs\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:36:58.122363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.122378 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4daadd53-1e04-4b2e-a073-b7dda180f82f-dshm\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:36:58.251847 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.251757 2565 generic.go:356] "Generic (PLEG): container finished" podID="4daadd53-1e04-4b2e-a073-b7dda180f82f" containerID="05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a" exitCode=0 Jul 10 20:36:58.251847 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.251796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" event={"ID":"4daadd53-1e04-4b2e-a073-b7dda180f82f","Type":"ContainerDied","Data":"05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a"} Jul 10 20:36:58.252131 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.251848 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" event={"ID":"4daadd53-1e04-4b2e-a073-b7dda180f82f","Type":"ContainerDied","Data":"6a07ceafe7ff400b6087f24de0a0594ee9d2e85a21db86c844aa6c19bda45048"} Jul 10 20:36:58.252131 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.251807 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67" Jul 10 20:36:58.252131 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.251899 2565 scope.go:117] "RemoveContainer" containerID="05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a" Jul 10 20:36:58.262538 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.262516 2565 scope.go:117] "RemoveContainer" containerID="05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a" Jul 10 20:36:58.262897 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:36:58.262850 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a\": container with ID starting with 05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a not found: ID does not exist" containerID="05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a" Jul 10 20:36:58.263037 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.262903 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a"} err="failed to get container status \"05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a\": rpc error: code = NotFound desc = could not find container \"05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a\": container with ID starting with 05d605ab167a5ed3738b7b3512985b597469886fe80bb217b43b87e4e34dce2a not found: ID does not exist" Jul 10 20:36:58.269241 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.269209 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67"] Jul 10 20:36:58.271830 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:36:58.271792 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-02c11db1/e2e-lifecycle-model-02c11db1-kserve-7487ddccf5-vnq67"] Jul 10 20:37:00.042545 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:37:00.042462 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4daadd53-1e04-4b2e-a073-b7dda180f82f" path="/var/lib/kubelet/pods/4daadd53-1e04-4b2e-a073-b7dda180f82f/volumes" Jul 10 20:38:40.043322 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:40.043291 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:38:40.047873 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:40.047835 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:38:52.056502 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.056466 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t"] Jul 10 20:38:52.056926 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.056831 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4daadd53-1e04-4b2e-a073-b7dda180f82f" containerName="main" Jul 10 20:38:52.056926 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.056842 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4daadd53-1e04-4b2e-a073-b7dda180f82f" containerName="main" Jul 10 20:38:52.057000 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.056943 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4daadd53-1e04-4b2e-a073-b7dda180f82f" containerName="main" Jul 10 20:38:52.059896 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.059875 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.063118 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.063092 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-58c4c1\"/\"openshift-service-ca.crt\"" Jul 10 20:38:52.063260 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.063141 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-58c4c1\"/\"kube-root-ca.crt\"" Jul 10 20:38:52.063909 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.063885 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-58c4c1\"/\"test-model-58c4c1-kserve-self-signed-certs\"" Jul 10 20:38:52.064034 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.063926 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-58c4c1\"/\"default-dockercfg-fkwcj\"" Jul 10 20:38:52.072679 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.072650 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t"] Jul 10 20:38:52.107053 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.107018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-dshm\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.107243 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.107069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-model-cache\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.107243 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.107101 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-tmp-dir\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.107243 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.107139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtr2\" (UniqueName: \"kubernetes.io/projected/da065329-dbf4-4f6c-be26-a53d9e73a560-kube-api-access-5jtr2\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.107243 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.107162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-home\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.107374 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.107282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da065329-dbf4-4f6c-be26-a53d9e73a560-tls-certs\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.207939 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.207897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-dshm\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.208135 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.207949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-model-cache\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.208135 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.207971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-tmp-dir\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.208135 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.208008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtr2\" (UniqueName: \"kubernetes.io/projected/da065329-dbf4-4f6c-be26-a53d9e73a560-kube-api-access-5jtr2\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.208135 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.208026 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-home\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.208135 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.208078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da065329-dbf4-4f6c-be26-a53d9e73a560-tls-certs\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.208428 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.208404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-model-cache\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.208485 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.208449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-home\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.208525 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.208478 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-tmp-dir\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.210300 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.210279 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-dshm\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.210580 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.210562 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da065329-dbf4-4f6c-be26-a53d9e73a560-tls-certs\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.216269 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.216235 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtr2\" (UniqueName: \"kubernetes.io/projected/da065329-dbf4-4f6c-be26-a53d9e73a560-kube-api-access-5jtr2\") pod \"test-model-58c4c1-kserve-5cbfcf9445-rjv6t\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.372074 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.371980 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:38:52.512599 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.512516 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t"] Jul 10 20:38:52.514964 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:38:52.514933 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda065329_dbf4_4f6c_be26_a53d9e73a560.slice/crio-0efcf45ea3e4ea77007e56f8b617c2bf24ce9b9622e1c825b79a587b689f358f WatchSource:0}: Error finding container 0efcf45ea3e4ea77007e56f8b617c2bf24ce9b9622e1c825b79a587b689f358f: Status 404 returned error can't find the container with id 0efcf45ea3e4ea77007e56f8b617c2bf24ce9b9622e1c825b79a587b689f358f Jul 10 20:38:52.806642 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.806601 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" event={"ID":"da065329-dbf4-4f6c-be26-a53d9e73a560","Type":"ContainerStarted","Data":"9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1"} Jul 10 20:38:52.806889 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.806653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" event={"ID":"da065329-dbf4-4f6c-be26-a53d9e73a560","Type":"ContainerStarted","Data":"0efcf45ea3e4ea77007e56f8b617c2bf24ce9b9622e1c825b79a587b689f358f"} Jul 10 20:38:52.827652 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:38:52.827592 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" podStartSLOduration=0.827573913 podStartE2EDuration="827.573913ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:38:52.825931199 +0000 UTC m=+1813.298900526" watchObservedRunningTime="2026-07-10 20:38:52.827573913 +0000 UTC m=+1813.300543239" Jul 10 20:39:02.372285 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:02.372244 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:39:02.372819 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:02.372305 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:39:02.385957 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:02.385924 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:39:02.872305 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:02.872277 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:39:30.965726 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:30.965693 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t"] Jul 10 20:39:45.988826 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:45.988786 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" podUID="da065329-dbf4-4f6c-be26-a53d9e73a560" containerName="main" containerID="cri-o://9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1" gracePeriod=45 Jul 10 20:39:46.247876 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.247794 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:39:46.431971 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.431931 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jtr2\" (UniqueName: \"kubernetes.io/projected/da065329-dbf4-4f6c-be26-a53d9e73a560-kube-api-access-5jtr2\") pod \"da065329-dbf4-4f6c-be26-a53d9e73a560\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " Jul 10 20:39:46.432146 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.431985 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-home\") pod \"da065329-dbf4-4f6c-be26-a53d9e73a560\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " Jul 10 20:39:46.432146 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432029 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-model-cache\") pod \"da065329-dbf4-4f6c-be26-a53d9e73a560\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " Jul 10 20:39:46.432146 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432048 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-tmp-dir\") pod \"da065329-dbf4-4f6c-be26-a53d9e73a560\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " Jul 10 20:39:46.432146 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432069 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da065329-dbf4-4f6c-be26-a53d9e73a560-tls-certs\") pod \"da065329-dbf4-4f6c-be26-a53d9e73a560\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " Jul 10 20:39:46.432146 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432103 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-dshm\") pod \"da065329-dbf4-4f6c-be26-a53d9e73a560\" (UID: \"da065329-dbf4-4f6c-be26-a53d9e73a560\") " Jul 10 20:39:46.432396 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432311 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da065329-dbf4-4f6c-be26-a53d9e73a560" (UID: "da065329-dbf4-4f6c-be26-a53d9e73a560"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:39:46.432396 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432332 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-model-cache" (OuterVolumeSpecName: "model-cache") pod "da065329-dbf4-4f6c-be26-a53d9e73a560" (UID: "da065329-dbf4-4f6c-be26-a53d9e73a560"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:39:46.432396 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432348 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-home" (OuterVolumeSpecName: "home") pod "da065329-dbf4-4f6c-be26-a53d9e73a560" (UID: "da065329-dbf4-4f6c-be26-a53d9e73a560"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:39:46.432542 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432463 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-home\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:39:46.432542 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432476 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-model-cache\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:39:46.432542 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.432487 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-tmp-dir\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:39:46.434409 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.434385 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-dshm" (OuterVolumeSpecName: "dshm") pod "da065329-dbf4-4f6c-be26-a53d9e73a560" (UID: "da065329-dbf4-4f6c-be26-a53d9e73a560"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:39:46.434505 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.434412 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da065329-dbf4-4f6c-be26-a53d9e73a560-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da065329-dbf4-4f6c-be26-a53d9e73a560" (UID: "da065329-dbf4-4f6c-be26-a53d9e73a560"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:39:46.434505 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.434491 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da065329-dbf4-4f6c-be26-a53d9e73a560-kube-api-access-5jtr2" (OuterVolumeSpecName: "kube-api-access-5jtr2") pod "da065329-dbf4-4f6c-be26-a53d9e73a560" (UID: "da065329-dbf4-4f6c-be26-a53d9e73a560"). InnerVolumeSpecName "kube-api-access-5jtr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:39:46.533065 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.532973 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jtr2\" (UniqueName: \"kubernetes.io/projected/da065329-dbf4-4f6c-be26-a53d9e73a560-kube-api-access-5jtr2\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:39:46.533065 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.533007 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da065329-dbf4-4f6c-be26-a53d9e73a560-tls-certs\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:39:46.533065 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:46.533016 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da065329-dbf4-4f6c-be26-a53d9e73a560-dshm\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:39:47.067180 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:47.067138 2565 generic.go:356] "Generic (PLEG): container finished" podID="da065329-dbf4-4f6c-be26-a53d9e73a560" containerID="9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1" exitCode=0 Jul 10 20:39:47.067573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:47.067205 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" Jul 10 20:39:47.067573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:47.067236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" event={"ID":"da065329-dbf4-4f6c-be26-a53d9e73a560","Type":"ContainerDied","Data":"9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1"} Jul 10 20:39:47.067573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:47.067294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t" event={"ID":"da065329-dbf4-4f6c-be26-a53d9e73a560","Type":"ContainerDied","Data":"0efcf45ea3e4ea77007e56f8b617c2bf24ce9b9622e1c825b79a587b689f358f"} Jul 10 20:39:47.067573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:47.067313 2565 scope.go:117] "RemoveContainer" containerID="9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1" Jul 10 20:39:47.077739 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:47.077717 2565 scope.go:117] "RemoveContainer" containerID="9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1" Jul 10 20:39:47.078112 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:39:47.078089 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1\": container with ID starting with 9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1 not found: ID does not exist" containerID="9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1" Jul 10 20:39:47.078349 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:47.078122 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1"} err="failed to get container status \"9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1\": rpc error: code = NotFound desc = could not find container \"9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1\": container with ID starting with 9aca2b654a771105d6a51dd138bc15b099b3d8b96ab42b47b36711178998d6c1 not found: ID does not exist" Jul 10 20:39:47.089447 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:47.089416 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t"] Jul 10 20:39:47.095206 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:47.095173 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-58c4c1/test-model-58c4c1-kserve-5cbfcf9445-rjv6t"] Jul 10 20:39:48.041057 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:39:48.041019 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da065329-dbf4-4f6c-be26-a53d9e73a560" path="/var/lib/kubelet/pods/da065329-dbf4-4f6c-be26-a53d9e73a560/volumes" Jul 10 20:41:31.845881 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.845773 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2"] Jul 10 20:41:31.846309 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.846207 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da065329-dbf4-4f6c-be26-a53d9e73a560" containerName="main" Jul 10 20:41:31.846309 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.846220 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da065329-dbf4-4f6c-be26-a53d9e73a560" containerName="main" Jul 10 20:41:31.846309 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.846282 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da065329-dbf4-4f6c-be26-a53d9e73a560" containerName="main" Jul 10 20:41:31.851984 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.850436 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:31.854573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.854550 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-75e636\"/\"default-dockercfg-qlpc9\"" Jul 10 20:41:31.855723 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.855694 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-75e636\"/\"auth-test-model-75e636-kserve-self-signed-certs\"" Jul 10 20:41:31.855723 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.855702 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-75e636\"/\"openshift-service-ca.crt\"" Jul 10 20:41:31.855974 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.855901 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-75e636\"/\"kube-root-ca.crt\"" Jul 10 20:41:31.858487 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.858464 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2"] Jul 10 20:41:31.954101 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.954056 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tls-certs\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:31.954270 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.954173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tmp-dir\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:31.954270 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.954234 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-home\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:31.954270 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.954254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff6rc\" (UniqueName: \"kubernetes.io/projected/cdf22d58-50c7-4974-bf3c-984cdc838fe5-kube-api-access-ff6rc\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:31.954373 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.954276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-dshm\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:31.954373 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:31.954316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-model-cache\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.054938 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.054898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-model-cache\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.055084 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.054960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tls-certs\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.055084 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.055038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tmp-dir\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.055084 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.055080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-home\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.055197 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.055106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff6rc\" (UniqueName: \"kubernetes.io/projected/cdf22d58-50c7-4974-bf3c-984cdc838fe5-kube-api-access-ff6rc\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.055197 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.055141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-dshm\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.055370 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.055350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-model-cache\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.055445 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.055424 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-home\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.055487 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.055453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tmp-dir\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.057484 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.057452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-dshm\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.057739 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.057654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tls-certs\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.064492 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.064465 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff6rc\" (UniqueName: \"kubernetes.io/projected/cdf22d58-50c7-4974-bf3c-984cdc838fe5-kube-api-access-ff6rc\") pod \"auth-test-model-75e636-kserve-76b47855ff-fmpq2\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.162034 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.161995 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:32.303267 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.303239 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2"] Jul 10 20:41:32.305894 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:41:32.305841 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdf22d58_50c7_4974_bf3c_984cdc838fe5.slice/crio-5e873aed9e3de13412af18791446492849decd4a255686d781232de2a9fa3c3c WatchSource:0}: Error finding container 5e873aed9e3de13412af18791446492849decd4a255686d781232de2a9fa3c3c: Status 404 returned error can't find the container with id 5e873aed9e3de13412af18791446492849decd4a255686d781232de2a9fa3c3c Jul 10 20:41:32.575192 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.575101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" event={"ID":"cdf22d58-50c7-4974-bf3c-984cdc838fe5","Type":"ContainerStarted","Data":"242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a"} Jul 10 20:41:32.575192 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.575151 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" event={"ID":"cdf22d58-50c7-4974-bf3c-984cdc838fe5","Type":"ContainerStarted","Data":"5e873aed9e3de13412af18791446492849decd4a255686d781232de2a9fa3c3c"} Jul 10 20:41:32.592765 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:32.592707 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" podStartSLOduration=1.592689242 podStartE2EDuration="1.592689242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:41:32.59221681 +0000 UTC m=+1973.065186152" watchObservedRunningTime="2026-07-10 20:41:32.592689242 +0000 UTC m=+1973.065658567" Jul 10 20:41:42.162319 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:42.162277 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:42.162728 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:42.162334 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:42.179597 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:42.179567 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:41:42.634155 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:41:42.634129 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:42:06.441535 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.441496 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7"] Jul 10 20:42:06.445312 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.445289 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.447743 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.447718 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"kube-root-ca.crt\"" Jul 10 20:42:06.448546 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.448532 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"default-dockercfg-bb8nm\"" Jul 10 20:42:06.448644 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.448596 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"openshift-service-ca.crt\"" Jul 10 20:42:06.448857 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.448844 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"auth-test-model-1b46b3-kserve-self-signed-certs\"" Jul 10 20:42:06.453907 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.453883 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7"] Jul 10 20:42:06.473151 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.473116 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-model-cache\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.473151 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.473155 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tls-certs\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.473428 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.473185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tmp-dir\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.473428 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.473303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-home\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.473428 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.473385 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmdl\" (UniqueName: \"kubernetes.io/projected/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-kube-api-access-8zmdl\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.473574 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.473441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-dshm\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.574926 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.574884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-model-cache\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.574926 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.574921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tls-certs\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.575185 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.574944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tmp-dir\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.575185 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.575002 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-home\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.575185 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.575055 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmdl\" (UniqueName: \"kubernetes.io/projected/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-kube-api-access-8zmdl\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.575185 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.575095 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-dshm\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.575395 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.575352 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-model-cache\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.575449 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.575392 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tmp-dir\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.575644 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.575626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-home\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.577464 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.577441 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-dshm\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.577790 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.577773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tls-certs\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.583848 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.583823 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmdl\" (UniqueName: \"kubernetes.io/projected/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-kube-api-access-8zmdl\") pod \"auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.755506 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.755404 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:06.912718 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:06.912695 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7"] Jul 10 20:42:06.915048 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:42:06.915010 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48c55469_efb4_4aaa_b9a3_2cd226dcaa45.slice/crio-6fddc9ed98ef86b3f315db5e91e5c616f5c9c71ea6682f3b432c6a3b8faf861a WatchSource:0}: Error finding container 6fddc9ed98ef86b3f315db5e91e5c616f5c9c71ea6682f3b432c6a3b8faf861a: Status 404 returned error can't find the container with id 6fddc9ed98ef86b3f315db5e91e5c616f5c9c71ea6682f3b432c6a3b8faf861a Jul 10 20:42:07.744710 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:07.744667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" event={"ID":"48c55469-efb4-4aaa-b9a3-2cd226dcaa45","Type":"ContainerStarted","Data":"69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0"} Jul 10 20:42:07.744710 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:07.744713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" event={"ID":"48c55469-efb4-4aaa-b9a3-2cd226dcaa45","Type":"ContainerStarted","Data":"6fddc9ed98ef86b3f315db5e91e5c616f5c9c71ea6682f3b432c6a3b8faf861a"} Jul 10 20:42:07.762413 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:07.762355 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" podStartSLOduration=1.762336848 podStartE2EDuration="1.762336848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:42:07.760954712 +0000 UTC m=+2008.233924040" watchObservedRunningTime="2026-07-10 20:42:07.762336848 +0000 UTC m=+2008.235306174" Jul 10 20:42:16.755712 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:16.755667 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:16.756156 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:16.755812 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:16.768454 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:16.768425 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:42:16.800580 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:42:16.800546 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:43:40.074337 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:43:40.074305 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:43:40.079466 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:43:40.079442 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v8bxg_88851b04-9c40-4f05-a3f2-7d62c45e32db/ovn-acl-logging/0.log" Jul 10 20:44:16.718816 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:16.718775 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2"] Jul 10 20:44:17.651722 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:17.651683 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7"] Jul 10 20:44:31.739789 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:31.739700 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" podUID="cdf22d58-50c7-4974-bf3c-984cdc838fe5" containerName="main" containerID="cri-o://242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a" gracePeriod=45 Jul 10 20:44:31.996212 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:31.996139 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:44:32.103633 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.103602 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff6rc\" (UniqueName: \"kubernetes.io/projected/cdf22d58-50c7-4974-bf3c-984cdc838fe5-kube-api-access-ff6rc\") pod \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " Jul 10 20:44:32.103828 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.103655 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tls-certs\") pod \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " Jul 10 20:44:32.103828 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.103679 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tmp-dir\") pod \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " Jul 10 20:44:32.103828 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.103714 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-home\") pod \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " Jul 10 20:44:32.103828 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.103743 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-dshm\") pod \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " Jul 10 20:44:32.103828 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.103784 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-model-cache\") pod \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\" (UID: \"cdf22d58-50c7-4974-bf3c-984cdc838fe5\") " Jul 10 20:44:32.104134 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.104004 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdf22d58-50c7-4974-bf3c-984cdc838fe5" (UID: "cdf22d58-50c7-4974-bf3c-984cdc838fe5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:44:32.104134 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.104097 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-home" (OuterVolumeSpecName: "home") pod "cdf22d58-50c7-4974-bf3c-984cdc838fe5" (UID: "cdf22d58-50c7-4974-bf3c-984cdc838fe5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:44:32.104241 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.104187 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tmp-dir\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:32.104241 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.104194 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdf22d58-50c7-4974-bf3c-984cdc838fe5" (UID: "cdf22d58-50c7-4974-bf3c-984cdc838fe5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:44:32.106138 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.106115 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdf22d58-50c7-4974-bf3c-984cdc838fe5" (UID: "cdf22d58-50c7-4974-bf3c-984cdc838fe5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:44:32.106224 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.106182 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf22d58-50c7-4974-bf3c-984cdc838fe5-kube-api-access-ff6rc" (OuterVolumeSpecName: "kube-api-access-ff6rc") pod "cdf22d58-50c7-4974-bf3c-984cdc838fe5" (UID: "cdf22d58-50c7-4974-bf3c-984cdc838fe5"). InnerVolumeSpecName "kube-api-access-ff6rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:44:32.106262 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.106217 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-dshm" (OuterVolumeSpecName: "dshm") pod "cdf22d58-50c7-4974-bf3c-984cdc838fe5" (UID: "cdf22d58-50c7-4974-bf3c-984cdc838fe5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:44:32.205656 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.205616 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-dshm\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:32.205656 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.205650 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-model-cache\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:32.205656 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.205665 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff6rc\" (UniqueName: \"kubernetes.io/projected/cdf22d58-50c7-4974-bf3c-984cdc838fe5-kube-api-access-ff6rc\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:32.205958 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.205673 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf22d58-50c7-4974-bf3c-984cdc838fe5-tls-certs\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:32.205958 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.205682 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf22d58-50c7-4974-bf3c-984cdc838fe5-home\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:32.459289 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.459247 2565 generic.go:356] "Generic (PLEG): container finished" podID="cdf22d58-50c7-4974-bf3c-984cdc838fe5" containerID="242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a" exitCode=0 Jul 10 20:44:32.459289 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.459289 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" Jul 10 20:44:32.459573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.459329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" event={"ID":"cdf22d58-50c7-4974-bf3c-984cdc838fe5","Type":"ContainerDied","Data":"242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a"} Jul 10 20:44:32.459573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.459386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2" event={"ID":"cdf22d58-50c7-4974-bf3c-984cdc838fe5","Type":"ContainerDied","Data":"5e873aed9e3de13412af18791446492849decd4a255686d781232de2a9fa3c3c"} Jul 10 20:44:32.459573 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.459409 2565 scope.go:117] "RemoveContainer" containerID="242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a" Jul 10 20:44:32.469783 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.469761 2565 scope.go:117] "RemoveContainer" containerID="242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a" Jul 10 20:44:32.470122 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:44:32.470096 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a\": container with ID starting with 242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a not found: ID does not exist" containerID="242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a" Jul 10 20:44:32.470204 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.470135 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a"} err="failed to get container status \"242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a\": rpc error: code = NotFound desc = could not find container \"242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a\": container with ID starting with 242a75b8df1fb897264b973942da331a0713db7d50669fe55eb531ef081acc2a not found: ID does not exist" Jul 10 20:44:32.480654 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.480623 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2"] Jul 10 20:44:32.484321 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.484292 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/auth-test-model-75e636-kserve-76b47855ff-fmpq2"] Jul 10 20:44:32.675885 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.675818 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" podUID="48c55469-efb4-4aaa-b9a3-2cd226dcaa45" containerName="main" containerID="cri-o://69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0" gracePeriod=45 Jul 10 20:44:32.924040 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:32.924013 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:44:33.014283 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014187 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tmp-dir\") pod \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " Jul 10 20:44:33.014283 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014240 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-dshm\") pod \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " Jul 10 20:44:33.014283 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014272 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-home\") pod \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " Jul 10 20:44:33.014565 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014296 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-model-cache\") pod \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " Jul 10 20:44:33.014565 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014322 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tls-certs\") pod \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " Jul 10 20:44:33.014565 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014390 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zmdl\" (UniqueName: \"kubernetes.io/projected/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-kube-api-access-8zmdl\") pod \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\" (UID: \"48c55469-efb4-4aaa-b9a3-2cd226dcaa45\") " Jul 10 20:44:33.014565 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014519 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48c55469-efb4-4aaa-b9a3-2cd226dcaa45" (UID: "48c55469-efb4-4aaa-b9a3-2cd226dcaa45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:44:33.014565 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014532 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-model-cache" (OuterVolumeSpecName: "model-cache") pod "48c55469-efb4-4aaa-b9a3-2cd226dcaa45" (UID: "48c55469-efb4-4aaa-b9a3-2cd226dcaa45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:44:33.014800 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014565 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-home" (OuterVolumeSpecName: "home") pod "48c55469-efb4-4aaa-b9a3-2cd226dcaa45" (UID: "48c55469-efb4-4aaa-b9a3-2cd226dcaa45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:44:33.014800 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014716 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tmp-dir\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:33.014800 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014730 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-home\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:33.014800 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.014739 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-model-cache\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:33.016711 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.016684 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-dshm" (OuterVolumeSpecName: "dshm") pod "48c55469-efb4-4aaa-b9a3-2cd226dcaa45" (UID: "48c55469-efb4-4aaa-b9a3-2cd226dcaa45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:44:33.016811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.016705 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48c55469-efb4-4aaa-b9a3-2cd226dcaa45" (UID: "48c55469-efb4-4aaa-b9a3-2cd226dcaa45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:44:33.016811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.016721 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-kube-api-access-8zmdl" (OuterVolumeSpecName: "kube-api-access-8zmdl") pod "48c55469-efb4-4aaa-b9a3-2cd226dcaa45" (UID: "48c55469-efb4-4aaa-b9a3-2cd226dcaa45"). InnerVolumeSpecName "kube-api-access-8zmdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:44:33.116039 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.115992 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-dshm\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:33.116039 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.116027 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-tls-certs\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:33.116039 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.116038 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zmdl\" (UniqueName: \"kubernetes.io/projected/48c55469-efb4-4aaa-b9a3-2cd226dcaa45-kube-api-access-8zmdl\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:44:33.466529 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.466490 2565 generic.go:356] "Generic (PLEG): container finished" podID="48c55469-efb4-4aaa-b9a3-2cd226dcaa45" containerID="69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0" exitCode=0 Jul 10 20:44:33.466723 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.466547 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" Jul 10 20:44:33.466723 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.466550 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" event={"ID":"48c55469-efb4-4aaa-b9a3-2cd226dcaa45","Type":"ContainerDied","Data":"69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0"} Jul 10 20:44:33.466723 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.466666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7" event={"ID":"48c55469-efb4-4aaa-b9a3-2cd226dcaa45","Type":"ContainerDied","Data":"6fddc9ed98ef86b3f315db5e91e5c616f5c9c71ea6682f3b432c6a3b8faf861a"} Jul 10 20:44:33.466723 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.466684 2565 scope.go:117] "RemoveContainer" containerID="69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0" Jul 10 20:44:33.476776 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.476749 2565 scope.go:117] "RemoveContainer" containerID="69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0" Jul 10 20:44:33.477123 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:44:33.477101 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0\": container with ID starting with 69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0 not found: ID does not exist" containerID="69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0" Jul 10 20:44:33.477214 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.477130 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0"} err="failed to get container status \"69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0\": rpc error: code = NotFound desc = could not find container \"69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0\": container with ID starting with 69211c2a5ab088daddaa4d1da33225c6fdf28d6ae44ceecce63daa10c30750d0 not found: ID does not exist" Jul 10 20:44:33.489574 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.489539 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7"] Jul 10 20:44:33.492809 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:33.492779 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/auth-test-model-1b46b3-kserve-d89fc5f55-7tcm7"] Jul 10 20:44:34.041881 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:34.041830 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c55469-efb4-4aaa-b9a3-2cd226dcaa45" path="/var/lib/kubelet/pods/48c55469-efb4-4aaa-b9a3-2cd226dcaa45/volumes" Jul 10 20:44:34.042287 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:44:34.042272 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdf22d58-50c7-4974-bf3c-984cdc838fe5" path="/var/lib/kubelet/pods/cdf22d58-50c7-4974-bf3c-984cdc838fe5/volumes" Jul 10 20:45:08.746151 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.746117 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7"] Jul 10 20:45:08.746593 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.746467 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf22d58-50c7-4974-bf3c-984cdc838fe5" containerName="main" Jul 10 20:45:08.746593 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.746478 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf22d58-50c7-4974-bf3c-984cdc838fe5" containerName="main" Jul 10 20:45:08.746593 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.746495 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c55469-efb4-4aaa-b9a3-2cd226dcaa45" containerName="main" Jul 10 20:45:08.746593 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.746500 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c55469-efb4-4aaa-b9a3-2cd226dcaa45" containerName="main" Jul 10 20:45:08.746593 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.746554 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c55469-efb4-4aaa-b9a3-2cd226dcaa45" containerName="main" Jul 10 20:45:08.746593 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.746567 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdf22d58-50c7-4974-bf3c-984cdc838fe5" containerName="main" Jul 10 20:45:08.750000 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.749975 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.753525 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.753482 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"openshift-service-ca.crt\"" Jul 10 20:45:08.753849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.753539 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"sub-test-model-1b46b3-kserve-self-signed-certs\"" Jul 10 20:45:08.753849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.753493 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"kube-root-ca.crt\"" Jul 10 20:45:08.753849 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.753544 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"default-dockercfg-bb8nm\"" Jul 10 20:45:08.759393 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.759361 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7"] Jul 10 20:45:08.853502 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.853468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-model-cache\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.853684 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.853526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-dshm\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.853684 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.853552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-home\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.853684 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.853574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pps27\" (UniqueName: \"kubernetes.io/projected/dd31a207-5112-4f29-8a69-830b079f23ed-kube-api-access-pps27\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.853788 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.853678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd31a207-5112-4f29-8a69-830b079f23ed-tls-certs\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.853788 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.853716 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-tmp-dir\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.954891 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.954803 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-model-cache\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.954891 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.954911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-dshm\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.955124 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.954932 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-home\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.955124 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.954954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pps27\" (UniqueName: \"kubernetes.io/projected/dd31a207-5112-4f29-8a69-830b079f23ed-kube-api-access-pps27\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.955124 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.955002 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd31a207-5112-4f29-8a69-830b079f23ed-tls-certs\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.955124 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.955034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-tmp-dir\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.955332 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.955293 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-model-cache\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.955376 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.955342 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-home\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.955417 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.955386 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-tmp-dir\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.957316 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.957297 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-dshm\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.957692 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.957672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd31a207-5112-4f29-8a69-830b079f23ed-tls-certs\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:08.964504 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:08.964469 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pps27\" (UniqueName: \"kubernetes.io/projected/dd31a207-5112-4f29-8a69-830b079f23ed-kube-api-access-pps27\") pod \"sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:09.062463 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:09.062348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:09.231783 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:09.231758 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7"] Jul 10 20:45:09.234806 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:45:09.234733 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd31a207_5112_4f29_8a69_830b079f23ed.slice/crio-fffcb7e7a0d0cadb8f3e5f333cf47e2ff7675d5d1b160306c26af89b0743b655 WatchSource:0}: Error finding container fffcb7e7a0d0cadb8f3e5f333cf47e2ff7675d5d1b160306c26af89b0743b655: Status 404 returned error can't find the container with id fffcb7e7a0d0cadb8f3e5f333cf47e2ff7675d5d1b160306c26af89b0743b655 Jul 10 20:45:09.636067 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:09.636023 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" event={"ID":"dd31a207-5112-4f29-8a69-830b079f23ed","Type":"ContainerStarted","Data":"bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982"} Jul 10 20:45:09.636067 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:09.636074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" event={"ID":"dd31a207-5112-4f29-8a69-830b079f23ed","Type":"ContainerStarted","Data":"fffcb7e7a0d0cadb8f3e5f333cf47e2ff7675d5d1b160306c26af89b0743b655"} Jul 10 20:45:09.654878 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:09.654796 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" podStartSLOduration=1.654780251 podStartE2EDuration="1.654780251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:45:09.652989767 +0000 UTC m=+2190.125959095" watchObservedRunningTime="2026-07-10 20:45:09.654780251 +0000 UTC m=+2190.127749595" Jul 10 20:45:19.063076 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:19.063024 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:19.063076 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:19.063096 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:19.075848 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:19.075817 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:19.701982 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:19.701950 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:44.440585 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:44.440552 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7"] Jul 10 20:45:53.035695 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.035651 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn"] Jul 10 20:45:53.039242 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.039216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.042656 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.042628 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-75e636\"/\"openshift-service-ca.crt\"" Jul 10 20:45:53.042656 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.042628 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-75e636\"/\"kube-root-ca.crt\"" Jul 10 20:45:53.042914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.042645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-75e636\"/\"default-dockercfg-qlpc9\"" Jul 10 20:45:53.042914 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.042702 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-75e636\"/\"rate-test-model-75e636-kserve-self-signed-certs\"" Jul 10 20:45:53.048319 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.048295 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn"] Jul 10 20:45:53.077637 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.077604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-dshm\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.077637 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.077641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-model-cache\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.077909 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.077664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-home\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.077909 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.077776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tls-certs\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.077909 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.077802 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tmp-dir\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.077909 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.077823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44p2v\" (UniqueName: \"kubernetes.io/projected/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-kube-api-access-44p2v\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.178479 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.178436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-home\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.178663 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.178533 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tls-certs\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.178663 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.178558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tmp-dir\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.178663 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.178582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44p2v\" (UniqueName: \"kubernetes.io/projected/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-kube-api-access-44p2v\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.178663 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.178622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-dshm\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.178663 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.178643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-model-cache\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.179042 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.179013 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-model-cache\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.179165 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.179066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tmp-dir\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.179165 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.179083 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-home\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.181103 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.181080 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-dshm\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.181353 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.181332 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tls-certs\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.188342 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.188309 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44p2v\" (UniqueName: \"kubernetes.io/projected/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-kube-api-access-44p2v\") pod \"rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.351138 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.351033 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:45:53.498448 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.498421 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn"] Jul 10 20:45:53.500913 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:45:53.500887 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f61a9c3_ceac_43a3_b0b3_4ca2113be733.slice/crio-5a4cd4db4981a1565cc9dd6f92aa3ae501b6149d63337a6880858a62dccd0fdb WatchSource:0}: Error finding container 5a4cd4db4981a1565cc9dd6f92aa3ae501b6149d63337a6880858a62dccd0fdb: Status 404 returned error can't find the container with id 5a4cd4db4981a1565cc9dd6f92aa3ae501b6149d63337a6880858a62dccd0fdb Jul 10 20:45:53.865881 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.865816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" event={"ID":"3f61a9c3-ceac-43a3-b0b3-4ca2113be733","Type":"ContainerStarted","Data":"fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9"} Jul 10 20:45:53.866091 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.865893 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" event={"ID":"3f61a9c3-ceac-43a3-b0b3-4ca2113be733","Type":"ContainerStarted","Data":"5a4cd4db4981a1565cc9dd6f92aa3ae501b6149d63337a6880858a62dccd0fdb"} Jul 10 20:45:53.882062 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:53.882006 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" podStartSLOduration=0.881988151 podStartE2EDuration="881.988151ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:45:53.880487502 +0000 UTC m=+2234.353456830" watchObservedRunningTime="2026-07-10 20:45:53.881988151 +0000 UTC m=+2234.354957476" Jul 10 20:45:59.462643 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.462592 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" podUID="dd31a207-5112-4f29-8a69-830b079f23ed" containerName="main" containerID="cri-o://bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982" gracePeriod=45 Jul 10 20:45:59.725216 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.725150 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:59.841840 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.841800 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-model-cache\") pod \"dd31a207-5112-4f29-8a69-830b079f23ed\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " Jul 10 20:45:59.842048 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.841883 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pps27\" (UniqueName: \"kubernetes.io/projected/dd31a207-5112-4f29-8a69-830b079f23ed-kube-api-access-pps27\") pod \"dd31a207-5112-4f29-8a69-830b079f23ed\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " Jul 10 20:45:59.842048 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.841931 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-tmp-dir\") pod \"dd31a207-5112-4f29-8a69-830b079f23ed\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " Jul 10 20:45:59.842048 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.841975 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-dshm\") pod \"dd31a207-5112-4f29-8a69-830b079f23ed\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " Jul 10 20:45:59.842048 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.842016 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-home\") pod \"dd31a207-5112-4f29-8a69-830b079f23ed\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " Jul 10 20:45:59.842318 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.842058 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd31a207-5112-4f29-8a69-830b079f23ed-tls-certs\") pod \"dd31a207-5112-4f29-8a69-830b079f23ed\" (UID: \"dd31a207-5112-4f29-8a69-830b079f23ed\") " Jul 10 20:45:59.842318 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.842191 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd31a207-5112-4f29-8a69-830b079f23ed" (UID: "dd31a207-5112-4f29-8a69-830b079f23ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:45:59.842318 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.842250 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd31a207-5112-4f29-8a69-830b079f23ed" (UID: "dd31a207-5112-4f29-8a69-830b079f23ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:45:59.842460 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.842348 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-home" (OuterVolumeSpecName: "home") pod "dd31a207-5112-4f29-8a69-830b079f23ed" (UID: "dd31a207-5112-4f29-8a69-830b079f23ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:45:59.842460 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.842401 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-tmp-dir\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:45:59.842460 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.842423 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-model-cache\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:45:59.844331 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.844296 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd31a207-5112-4f29-8a69-830b079f23ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd31a207-5112-4f29-8a69-830b079f23ed" (UID: "dd31a207-5112-4f29-8a69-830b079f23ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:45:59.844331 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.844320 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-dshm" (OuterVolumeSpecName: "dshm") pod "dd31a207-5112-4f29-8a69-830b079f23ed" (UID: "dd31a207-5112-4f29-8a69-830b079f23ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:45:59.844492 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.844400 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd31a207-5112-4f29-8a69-830b079f23ed-kube-api-access-pps27" (OuterVolumeSpecName: "kube-api-access-pps27") pod "dd31a207-5112-4f29-8a69-830b079f23ed" (UID: "dd31a207-5112-4f29-8a69-830b079f23ed"). InnerVolumeSpecName "kube-api-access-pps27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:45:59.898422 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.898389 2565 generic.go:356] "Generic (PLEG): container finished" podID="dd31a207-5112-4f29-8a69-830b079f23ed" containerID="bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982" exitCode=0 Jul 10 20:45:59.898626 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.898438 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" Jul 10 20:45:59.898626 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.898471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" event={"ID":"dd31a207-5112-4f29-8a69-830b079f23ed","Type":"ContainerDied","Data":"bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982"} Jul 10 20:45:59.898626 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.898523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" event={"ID":"dd31a207-5112-4f29-8a69-830b079f23ed","Type":"ContainerDied","Data":"fffcb7e7a0d0cadb8f3e5f333cf47e2ff7675d5d1b160306c26af89b0743b655"} Jul 10 20:45:59.898626 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.898540 2565 scope.go:117] "RemoveContainer" containerID="bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982" Jul 10 20:45:59.907816 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.907793 2565 scope.go:117] "RemoveContainer" containerID="bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982" Jul 10 20:45:59.908122 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:45:59.908097 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982\": container with ID starting with bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982 not found: ID does not exist" containerID="bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982" Jul 10 20:45:59.908194 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.908131 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982"} err="failed to get container status \"bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982\": rpc error: code = NotFound desc = could not find container \"bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982\": container with ID starting with bcde05de8e82de496198b22d27191b1ef8d255688f68272cd01f28dfebc31982 not found: ID does not exist" Jul 10 20:45:59.918782 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.918753 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7"] Jul 10 20:45:59.920466 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.920438 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7"] Jul 10 20:45:59.943941 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.943905 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-dshm\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:45:59.943941 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.943935 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd31a207-5112-4f29-8a69-830b079f23ed-home\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:45:59.943941 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.943946 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd31a207-5112-4f29-8a69-830b079f23ed-tls-certs\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:45:59.944172 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:45:59.943955 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pps27\" (UniqueName: \"kubernetes.io/projected/dd31a207-5112-4f29-8a69-830b079f23ed-kube-api-access-pps27\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:46:00.041379 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:00.041296 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd31a207-5112-4f29-8a69-830b079f23ed" path="/var/lib/kubelet/pods/dd31a207-5112-4f29-8a69-830b079f23ed/volumes" Jul 10 20:46:03.351391 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:03.351346 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:46:03.351796 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:03.351405 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:46:03.367299 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:03.367269 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:46:03.935879 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:03.935826 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:46:04.692091 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:04.692031 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-1b46b3/sub-test-model-1b46b3-kserve-75b6d9c47f-5snt7" podUID="dd31a207-5112-4f29-8a69-830b079f23ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 10 20:46:10.633220 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.633185 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld"] Jul 10 20:46:10.633709 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.633554 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd31a207-5112-4f29-8a69-830b079f23ed" containerName="main" Jul 10 20:46:10.633709 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.633565 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd31a207-5112-4f29-8a69-830b079f23ed" containerName="main" Jul 10 20:46:10.633709 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.633632 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd31a207-5112-4f29-8a69-830b079f23ed" containerName="main" Jul 10 20:46:10.638188 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.638161 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.641372 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.641334 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"kube-root-ca.crt\"" Jul 10 20:46:10.641372 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.641350 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"rate-test-model-1b46b3-kserve-self-signed-certs\"" Jul 10 20:46:10.641625 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.641350 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"default-dockercfg-bb8nm\"" Jul 10 20:46:10.641625 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.641446 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1b46b3\"/\"openshift-service-ca.crt\"" Jul 10 20:46:10.646905 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.646880 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld"] Jul 10 20:46:10.751317 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.751275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-tmp-dir\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.751317 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.751320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72svq\" (UniqueName: \"kubernetes.io/projected/67618922-3083-4505-974b-0aa020bf8ba7-kube-api-access-72svq\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.751537 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.751398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-home\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.751537 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.751438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67618922-3083-4505-974b-0aa020bf8ba7-tls-certs\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.751537 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.751500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-dshm\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.751537 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.751521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-model-cache\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.852856 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.852815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-model-cache\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.853061 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.852916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-tmp-dir\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.853061 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.852950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72svq\" (UniqueName: \"kubernetes.io/projected/67618922-3083-4505-974b-0aa020bf8ba7-kube-api-access-72svq\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.853061 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.852995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-home\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.853061 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.853015 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67618922-3083-4505-974b-0aa020bf8ba7-tls-certs\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.853302 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.853090 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-dshm\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.853358 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.853308 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-model-cache\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.853401 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.853351 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-tmp-dir\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.853435 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.853413 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-home\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.855416 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.855389 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-dshm\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.855645 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.855626 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67618922-3083-4505-974b-0aa020bf8ba7-tls-certs\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.861343 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.861314 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72svq\" (UniqueName: \"kubernetes.io/projected/67618922-3083-4505-974b-0aa020bf8ba7-kube-api-access-72svq\") pod \"rate-test-model-1b46b3-kserve-56d66947d7-5jlld\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:10.950223 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:10.950106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:11.093880 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:11.093823 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld"] Jul 10 20:46:11.097703 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:46:11.097667 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67618922_3083_4505_974b_0aa020bf8ba7.slice/crio-4de1f0a450cbd92aee9867eaaf9bf881b0b3197c5acde0285e0c893be824971d WatchSource:0}: Error finding container 4de1f0a450cbd92aee9867eaaf9bf881b0b3197c5acde0285e0c893be824971d: Status 404 returned error can't find the container with id 4de1f0a450cbd92aee9867eaaf9bf881b0b3197c5acde0285e0c893be824971d Jul 10 20:46:11.966384 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:11.966346 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" event={"ID":"67618922-3083-4505-974b-0aa020bf8ba7","Type":"ContainerStarted","Data":"a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663"} Jul 10 20:46:11.966384 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:11.966391 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" event={"ID":"67618922-3083-4505-974b-0aa020bf8ba7","Type":"ContainerStarted","Data":"4de1f0a450cbd92aee9867eaaf9bf881b0b3197c5acde0285e0c893be824971d"} Jul 10 20:46:11.984394 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:11.984337 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" podStartSLOduration=1.984318563 podStartE2EDuration="1.984318563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:46:11.98247124 +0000 UTC m=+2252.455440568" watchObservedRunningTime="2026-07-10 20:46:11.984318563 +0000 UTC m=+2252.457287888" Jul 10 20:46:20.950662 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:20.950624 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:20.951185 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:20.950684 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:20.965877 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:20.965830 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:21.024406 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:21.024376 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:46:57.523618 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:57.523583 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn"] Jul 10 20:46:58.542796 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:46:58.542761 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld"] Jul 10 20:47:12.544836 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.544784 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" podUID="3f61a9c3-ceac-43a3-b0b3-4ca2113be733" containerName="main" containerID="cri-o://fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9" gracePeriod=45 Jul 10 20:47:12.797522 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.797447 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:47:12.849841 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.849805 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-home\") pod \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " Jul 10 20:47:12.850049 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.849924 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tmp-dir\") pod \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " Jul 10 20:47:12.850049 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.849953 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-dshm\") pod \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " Jul 10 20:47:12.850049 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.849973 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tls-certs\") pod \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " Jul 10 20:47:12.850049 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.850001 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-model-cache\") pod \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " Jul 10 20:47:12.850263 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.850119 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-home" (OuterVolumeSpecName: "home") pod "3f61a9c3-ceac-43a3-b0b3-4ca2113be733" (UID: "3f61a9c3-ceac-43a3-b0b3-4ca2113be733"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:47:12.850263 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.850190 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44p2v\" (UniqueName: \"kubernetes.io/projected/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-kube-api-access-44p2v\") pod \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\" (UID: \"3f61a9c3-ceac-43a3-b0b3-4ca2113be733\") " Jul 10 20:47:12.850263 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.850220 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f61a9c3-ceac-43a3-b0b3-4ca2113be733" (UID: "3f61a9c3-ceac-43a3-b0b3-4ca2113be733"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:47:12.850420 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.850289 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f61a9c3-ceac-43a3-b0b3-4ca2113be733" (UID: "3f61a9c3-ceac-43a3-b0b3-4ca2113be733"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:47:12.850575 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.850556 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-home\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:12.850633 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.850580 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tmp-dir\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:12.850633 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.850595 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-model-cache\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:12.852320 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.852292 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f61a9c3-ceac-43a3-b0b3-4ca2113be733" (UID: "3f61a9c3-ceac-43a3-b0b3-4ca2113be733"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:47:12.852320 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.852308 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-dshm" (OuterVolumeSpecName: "dshm") pod "3f61a9c3-ceac-43a3-b0b3-4ca2113be733" (UID: "3f61a9c3-ceac-43a3-b0b3-4ca2113be733"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:47:12.852480 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.852319 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-kube-api-access-44p2v" (OuterVolumeSpecName: "kube-api-access-44p2v") pod "3f61a9c3-ceac-43a3-b0b3-4ca2113be733" (UID: "3f61a9c3-ceac-43a3-b0b3-4ca2113be733"). InnerVolumeSpecName "kube-api-access-44p2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:47:12.951450 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.951412 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44p2v\" (UniqueName: \"kubernetes.io/projected/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-kube-api-access-44p2v\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:12.951450 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.951443 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-dshm\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:12.951450 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:12.951452 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f61a9c3-ceac-43a3-b0b3-4ca2113be733-tls-certs\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:13.280820 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.280781 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f61a9c3-ceac-43a3-b0b3-4ca2113be733" containerID="fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9" exitCode=0 Jul 10 20:47:13.281051 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.280832 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" Jul 10 20:47:13.281051 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.280887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" event={"ID":"3f61a9c3-ceac-43a3-b0b3-4ca2113be733","Type":"ContainerDied","Data":"fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9"} Jul 10 20:47:13.281051 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.280944 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn" event={"ID":"3f61a9c3-ceac-43a3-b0b3-4ca2113be733","Type":"ContainerDied","Data":"5a4cd4db4981a1565cc9dd6f92aa3ae501b6149d63337a6880858a62dccd0fdb"} Jul 10 20:47:13.281051 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.280963 2565 scope.go:117] "RemoveContainer" containerID="fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9" Jul 10 20:47:13.290519 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.290500 2565 scope.go:117] "RemoveContainer" containerID="fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9" Jul 10 20:47:13.290821 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:47:13.290801 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9\": container with ID starting with fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9 not found: ID does not exist" containerID="fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9" Jul 10 20:47:13.290912 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.290836 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9"} err="failed to get container status \"fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9\": rpc error: code = NotFound desc = could not find container \"fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9\": container with ID starting with fc48b9309b23c55f4abfda79a2075880407949da05905a87d015aaf31b529dd9 not found: ID does not exist" Jul 10 20:47:13.300127 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.300097 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn"] Jul 10 20:47:13.305915 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.305883 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-75e636/rate-test-model-75e636-kserve-76cbf7bfc9-h7vkn"] Jul 10 20:47:13.566959 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.566846 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" podUID="67618922-3083-4505-974b-0aa020bf8ba7" containerName="main" containerID="cri-o://a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663" gracePeriod=45 Jul 10 20:47:13.830595 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.830518 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:47:13.960159 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960105 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-dshm\") pod \"67618922-3083-4505-974b-0aa020bf8ba7\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " Jul 10 20:47:13.960365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960182 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-tmp-dir\") pod \"67618922-3083-4505-974b-0aa020bf8ba7\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " Jul 10 20:47:13.960365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960239 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67618922-3083-4505-974b-0aa020bf8ba7-tls-certs\") pod \"67618922-3083-4505-974b-0aa020bf8ba7\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " Jul 10 20:47:13.960365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960283 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72svq\" (UniqueName: \"kubernetes.io/projected/67618922-3083-4505-974b-0aa020bf8ba7-kube-api-access-72svq\") pod \"67618922-3083-4505-974b-0aa020bf8ba7\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " Jul 10 20:47:13.960365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960311 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-model-cache\") pod \"67618922-3083-4505-974b-0aa020bf8ba7\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " Jul 10 20:47:13.960365 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960347 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-home\") pod \"67618922-3083-4505-974b-0aa020bf8ba7\" (UID: \"67618922-3083-4505-974b-0aa020bf8ba7\") " Jul 10 20:47:13.960643 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960449 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67618922-3083-4505-974b-0aa020bf8ba7" (UID: "67618922-3083-4505-974b-0aa020bf8ba7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:47:13.960643 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960591 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-model-cache" (OuterVolumeSpecName: "model-cache") pod "67618922-3083-4505-974b-0aa020bf8ba7" (UID: "67618922-3083-4505-974b-0aa020bf8ba7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:47:13.960643 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960624 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-home" (OuterVolumeSpecName: "home") pod "67618922-3083-4505-974b-0aa020bf8ba7" (UID: "67618922-3083-4505-974b-0aa020bf8ba7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:47:13.960811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960692 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-tmp-dir\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:13.960811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960709 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-model-cache\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:13.960811 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.960721 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-home\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:13.962563 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.962530 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67618922-3083-4505-974b-0aa020bf8ba7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67618922-3083-4505-974b-0aa020bf8ba7" (UID: "67618922-3083-4505-974b-0aa020bf8ba7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 20:47:13.962563 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.962545 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67618922-3083-4505-974b-0aa020bf8ba7-kube-api-access-72svq" (OuterVolumeSpecName: "kube-api-access-72svq") pod "67618922-3083-4505-974b-0aa020bf8ba7" (UID: "67618922-3083-4505-974b-0aa020bf8ba7"). InnerVolumeSpecName "kube-api-access-72svq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 20:47:13.962744 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:13.962730 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-dshm" (OuterVolumeSpecName: "dshm") pod "67618922-3083-4505-974b-0aa020bf8ba7" (UID: "67618922-3083-4505-974b-0aa020bf8ba7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 20:47:14.041821 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.041782 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f61a9c3-ceac-43a3-b0b3-4ca2113be733" path="/var/lib/kubelet/pods/3f61a9c3-ceac-43a3-b0b3-4ca2113be733/volumes" Jul 10 20:47:14.061624 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.061574 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67618922-3083-4505-974b-0aa020bf8ba7-tls-certs\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:14.061624 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.061616 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72svq\" (UniqueName: \"kubernetes.io/projected/67618922-3083-4505-974b-0aa020bf8ba7-kube-api-access-72svq\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:14.061624 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.061628 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67618922-3083-4505-974b-0aa020bf8ba7-dshm\") on node \"ip-10-0-143-108.ec2.internal\" DevicePath \"\"" Jul 10 20:47:14.288512 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.288478 2565 generic.go:356] "Generic (PLEG): container finished" podID="67618922-3083-4505-974b-0aa020bf8ba7" containerID="a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663" exitCode=0 Jul 10 20:47:14.288512 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.288516 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" Jul 10 20:47:14.288752 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.288563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" event={"ID":"67618922-3083-4505-974b-0aa020bf8ba7","Type":"ContainerDied","Data":"a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663"} Jul 10 20:47:14.288752 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.288605 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld" event={"ID":"67618922-3083-4505-974b-0aa020bf8ba7","Type":"ContainerDied","Data":"4de1f0a450cbd92aee9867eaaf9bf881b0b3197c5acde0285e0c893be824971d"} Jul 10 20:47:14.288752 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.288624 2565 scope.go:117] "RemoveContainer" containerID="a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663" Jul 10 20:47:14.297495 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.297470 2565 scope.go:117] "RemoveContainer" containerID="a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663" Jul 10 20:47:14.297802 ip-10-0-143-108 kubenswrapper[2565]: E0710 20:47:14.297782 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663\": container with ID starting with a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663 not found: ID does not exist" containerID="a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663" Jul 10 20:47:14.297928 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.297811 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663"} err="failed to get container status \"a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663\": rpc error: code = NotFound desc = could not find container \"a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663\": container with ID starting with a20d46df8d3af093590cecd69c0bcbb4f270a299ec6105fc5ce42377b958f663 not found: ID does not exist" Jul 10 20:47:14.304543 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.304514 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld"] Jul 10 20:47:14.309722 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:14.309692 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1b46b3/rate-test-model-1b46b3-kserve-56d66947d7-5jlld"] Jul 10 20:47:16.041420 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:16.041387 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67618922-3083-4505-974b-0aa020bf8ba7" path="/var/lib/kubelet/pods/67618922-3083-4505-974b-0aa020bf8ba7/volumes" Jul 10 20:47:32.059671 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:32.059591 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75f695b566-2f42g_2ff8108c-9747-4533-a7d9-2378bcfcb9c9/authorino/0.log" Jul 10 20:47:37.207262 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:37.207222 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rbztx_aad5355c-7671-4040-acae-49bfbfe60002/manager/0.log" Jul 10 20:47:37.442006 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:37.441976 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5767ffdff4-fv827_f53333fc-9d3a-4bbc-aca6-5d2d2cb36495/copy-manifests/0.log" Jul 10 20:47:37.449576 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:37.449547 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5767ffdff4-fv827_f53333fc-9d3a-4bbc-aca6-5d2d2cb36495/manager/0.log" Jul 10 20:47:39.298433 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:39.298392 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75f695b566-2f42g_2ff8108c-9747-4533-a7d9-2378bcfcb9c9/authorino/0.log" Jul 10 20:47:39.521296 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:39.521264 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pvgft_d4e5e042-290e-47bb-84af-567017e99f9c/manager/0.log" Jul 10 20:47:39.634091 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:39.634064 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pzjc8_d96ba105-89ab-4dc3-b139-28ded2c45713/kuadrant-console-plugin/0.log" Jul 10 20:47:39.880640 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:39.880598 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-b8rvc_cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d/manager/0.log" Jul 10 20:47:40.668264 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:40.668228 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74575f44df-7v8gk_9697b134-546f-4f75-9842-104a491b0e59/kube-auth-proxy/0.log" Jul 10 20:47:41.668924 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:41.668894 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dzlb4_6b0af3ed-e5c7-4bbf-b046-f37b9e0a72f8/main/0.log" Jul 10 20:47:41.887950 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:41.887914 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwbzpf_3945e298-4d86-4c35-bf50-bee3416b6c0e/main/0.log" Jul 10 20:47:42.003051 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:42.002967 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jbnth_a1fd351d-4151-4bdf-b2b5-f71bf71e123c/main/0.log" Jul 10 20:47:42.107911 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:42.107856 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-kb8f8_854748ac-51f1-401f-9b22-eb2ba1b2bc9f/main/0.log" Jul 10 20:47:49.179840 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:49.179805 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m7gpw_5e7a5331-206f-47f7-b6ee-237be41026bd/global-pull-secret-syncer/0.log" Jul 10 20:47:49.333037 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:49.333004 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l676f_46f45b82-e1e3-4b93-aca9-bbb73ac320e2/konnectivity-agent/0.log" Jul 10 20:47:49.406204 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:49.406170 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-108.ec2.internal_07aa0ca73f3d80d2650c468cb7b052ee/haproxy/0.log" Jul 10 20:47:54.871102 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:54.871064 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75f695b566-2f42g_2ff8108c-9747-4533-a7d9-2378bcfcb9c9/authorino/0.log" Jul 10 20:47:54.931375 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:54.931340 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pvgft_d4e5e042-290e-47bb-84af-567017e99f9c/manager/0.log" Jul 10 20:47:54.954709 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:54.954678 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pzjc8_d96ba105-89ab-4dc3-b139-28ded2c45713/kuadrant-console-plugin/0.log" Jul 10 20:47:55.228922 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:55.228882 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-b8rvc_cc1a48af-a1ed-492f-a7a2-b246dcb4bd3d/manager/0.log" Jul 10 20:47:57.386911 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.386879 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-65f947fd9c-7qwpw_6d6662c0-e3f0-423c-b5d5-1e6ec3d7ce10/metrics-server/0.log" Jul 10 20:47:57.411704 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.411620 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-xfhvg_06e4a08d-aacf-4197-86a4-e2d654ec33ba/monitoring-plugin/0.log" Jul 10 20:47:57.514001 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.513975 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtj9v_8adc0775-e750-4ace-92a9-b640c2d19417/node-exporter/0.log" Jul 10 20:47:57.536540 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.536508 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtj9v_8adc0775-e750-4ace-92a9-b640c2d19417/kube-rbac-proxy/0.log" Jul 10 20:47:57.557224 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.557194 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtj9v_8adc0775-e750-4ace-92a9-b640c2d19417/init-textfile/0.log" Jul 10 20:47:57.681069 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.680989 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-f8khk_95ac415d-4e61-42fa-8bb6-524cde061644/kube-rbac-proxy-main/0.log" Jul 10 20:47:57.701000 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.700972 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-f8khk_95ac415d-4e61-42fa-8bb6-524cde061644/kube-rbac-proxy-self/0.log" Jul 10 20:47:57.726999 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.726948 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-f8khk_95ac415d-4e61-42fa-8bb6-524cde061644/openshift-state-metrics/0.log" Jul 10 20:47:57.765347 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.765308 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5bff782-4bbe-45a3-8739-1645dc1d0617/prometheus/0.log" Jul 10 20:47:57.784703 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.784672 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5bff782-4bbe-45a3-8739-1645dc1d0617/config-reloader/0.log" Jul 10 20:47:57.818930 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.818889 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5bff782-4bbe-45a3-8739-1645dc1d0617/thanos-sidecar/0.log" Jul 10 20:47:57.836846 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.836813 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5bff782-4bbe-45a3-8739-1645dc1d0617/kube-rbac-proxy-web/0.log" Jul 10 20:47:57.863363 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.863336 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5bff782-4bbe-45a3-8739-1645dc1d0617/kube-rbac-proxy/0.log" Jul 10 20:47:57.884515 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.884486 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5bff782-4bbe-45a3-8739-1645dc1d0617/kube-rbac-proxy-thanos/0.log" Jul 10 20:47:57.905904 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.905849 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5bff782-4bbe-45a3-8739-1645dc1d0617/init-config-reloader/0.log" Jul 10 20:47:57.982433 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:57.982348 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-2pm69_7ef09f15-f370-46c9-ba22-d0ee4c0acd6a/prometheus-operator-admission-webhook/0.log" Jul 10 20:47:58.107979 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:58.107948 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69d5d74485-pj2qz_e2146e51-23dd-4d97-9a39-e15669b7db97/thanos-query/0.log" Jul 10 20:47:58.130640 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:58.130615 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69d5d74485-pj2qz_e2146e51-23dd-4d97-9a39-e15669b7db97/kube-rbac-proxy-web/0.log" Jul 10 20:47:58.150821 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:58.150788 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69d5d74485-pj2qz_e2146e51-23dd-4d97-9a39-e15669b7db97/kube-rbac-proxy/0.log" Jul 10 20:47:58.170888 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:58.170838 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69d5d74485-pj2qz_e2146e51-23dd-4d97-9a39-e15669b7db97/prom-label-proxy/0.log" Jul 10 20:47:58.194100 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:58.194069 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69d5d74485-pj2qz_e2146e51-23dd-4d97-9a39-e15669b7db97/kube-rbac-proxy-rules/0.log" Jul 10 20:47:58.214217 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:58.214179 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69d5d74485-pj2qz_e2146e51-23dd-4d97-9a39-e15669b7db97/kube-rbac-proxy-metrics/0.log" Jul 10 20:47:59.421047 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.421006 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9"] Jul 10 20:47:59.421646 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.421623 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f61a9c3-ceac-43a3-b0b3-4ca2113be733" containerName="main" Jul 10 20:47:59.421740 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.421650 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f61a9c3-ceac-43a3-b0b3-4ca2113be733" containerName="main" Jul 10 20:47:59.421740 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.421661 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67618922-3083-4505-974b-0aa020bf8ba7" containerName="main" Jul 10 20:47:59.421740 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.421670 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="67618922-3083-4505-974b-0aa020bf8ba7" containerName="main" Jul 10 20:47:59.421958 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.421757 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f61a9c3-ceac-43a3-b0b3-4ca2113be733" containerName="main" Jul 10 20:47:59.421958 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.421770 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="67618922-3083-4505-974b-0aa020bf8ba7" containerName="main" Jul 10 20:47:59.425030 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.425005 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.427245 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.427218 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xk8xm\"/\"default-dockercfg-jnk86\"" Jul 10 20:47:59.427467 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.427443 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xk8xm\"/\"openshift-service-ca.crt\"" Jul 10 20:47:59.428263 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.428245 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xk8xm\"/\"kube-root-ca.crt\"" Jul 10 20:47:59.432963 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.432941 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9"] Jul 10 20:47:59.499556 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.499513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-sys\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.499556 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.499557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-podres\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.499835 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.499588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55sq6\" (UniqueName: \"kubernetes.io/projected/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-kube-api-access-55sq6\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.499835 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.499712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-proc\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.499835 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.499787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-lib-modules\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.600448 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.600398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-sys\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.600448 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.600444 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-podres\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.600705 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.600470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55sq6\" (UniqueName: \"kubernetes.io/projected/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-kube-api-access-55sq6\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.600705 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.600535 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-sys\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.600705 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.600547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-proc\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.600705 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.600595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-podres\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.600705 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.600621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-lib-modules\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.600705 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.600605 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-proc\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.600948 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.600735 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-lib-modules\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.608775 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.608751 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55sq6\" (UniqueName: \"kubernetes.io/projected/ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2-kube-api-access-55sq6\") pod \"perf-node-gather-daemonset-btgl9\" (UID: \"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2\") " pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.735846 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.735764 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:47:59.872743 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:47:59.872700 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9"] Jul 10 20:47:59.875764 ip-10-0-143-108 kubenswrapper[2565]: W0710 20:47:59.875739 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podac9e6b10_0f42_4742_9d31_edc3d5c8c5b2.slice/crio-53c4250e69e5559e4cd56305fc5c3a0d14cb18a918e08d2d2801f7feb5c81e91 WatchSource:0}: Error finding container 53c4250e69e5559e4cd56305fc5c3a0d14cb18a918e08d2d2801f7feb5c81e91: Status 404 returned error can't find the container with id 53c4250e69e5559e4cd56305fc5c3a0d14cb18a918e08d2d2801f7feb5c81e91 Jul 10 20:48:00.456392 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:00.456356 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65cb9d569b-7pldd_6b3ae028-f71d-4c14-8c9b-1f3abcf9dfd5/console/0.log" Jul 10 20:48:00.513965 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:00.513925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" event={"ID":"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2","Type":"ContainerStarted","Data":"1161ebd745338f79ee95e10bb1430adf26ccf5ad923b28e8054d45c721cd1102"} Jul 10 20:48:00.514149 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:00.513988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" event={"ID":"ac9e6b10-0f42-4742-9d31-edc3d5c8c5b2","Type":"ContainerStarted","Data":"53c4250e69e5559e4cd56305fc5c3a0d14cb18a918e08d2d2801f7feb5c81e91"} Jul 10 20:48:00.514149 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:00.514030 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:48:00.529653 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:00.529600 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" podStartSLOduration=1.5295827910000002 podStartE2EDuration="1.529582791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:48:00.528897774 +0000 UTC m=+2361.001867092" watchObservedRunningTime="2026-07-10 20:48:00.529582791 +0000 UTC m=+2361.002552114" Jul 10 20:48:02.182767 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:02.182736 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kvgvk_80f7b86b-a451-477c-940a-0ab9fbcf8f62/dns/0.log" Jul 10 20:48:02.202585 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:02.202543 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kvgvk_80f7b86b-a451-477c-940a-0ab9fbcf8f62/kube-rbac-proxy/0.log" Jul 10 20:48:02.325093 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:02.325064 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-znc59_dd973a57-23d9-4ee5-985f-cae1defe06cf/dns-node-resolver/0.log" Jul 10 20:48:02.877794 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:02.877763 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5kkzz_93addf78-13b4-48cc-bb04-f4134b872483/node-ca/0.log" Jul 10 20:48:04.010800 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:04.010762 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74575f44df-7v8gk_9697b134-546f-4f75-9842-104a491b0e59/kube-auth-proxy/0.log" Jul 10 20:48:04.936330 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:04.936278 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rvf6x_9c782ed4-67a4-4d80-bcff-23e322671811/serve-healthcheck-canary/0.log" Jul 10 20:48:05.437054 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:05.437014 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bjfb5_411019d5-e164-41f1-91d8-ffd907a4319c/kube-rbac-proxy/0.log" Jul 10 20:48:05.459949 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:05.459910 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bjfb5_411019d5-e164-41f1-91d8-ffd907a4319c/exporter/0.log" Jul 10 20:48:05.485613 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:05.485573 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bjfb5_411019d5-e164-41f1-91d8-ffd907a4319c/extractor/0.log" Jul 10 20:48:06.529731 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:06.529701 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xk8xm/perf-node-gather-daemonset-btgl9" Jul 10 20:48:08.292828 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:08.292790 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rbztx_aad5355c-7671-4040-acae-49bfbfe60002/manager/0.log" Jul 10 20:48:08.374608 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:08.374579 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5767ffdff4-fv827_f53333fc-9d3a-4bbc-aca6-5d2d2cb36495/manager/0.log" Jul 10 20:48:08.396153 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:08.396079 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5767ffdff4-fv827_f53333fc-9d3a-4bbc-aca6-5d2d2cb36495/copy-manifests/0.log" Jul 10 20:48:09.877300 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:09.877272 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bd7b6bf4b-7lfpf_921861e7-7e09-47ea-a34f-2e876cbf2f4b/manager/0.log" Jul 10 20:48:09.938613 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:09.938583 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gnkgh_71d97f2e-7b89-4f83-8db2-a5398bb9d355/openshift-lws-operator/0.log" Jul 10 20:48:10.401802 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:10.401773 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74575f44df-7v8gk_9697b134-546f-4f75-9842-104a491b0e59/kube-auth-proxy/0.log" Jul 10 20:48:11.920207 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:11.920172 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74575f44df-7v8gk_9697b134-546f-4f75-9842-104a491b0e59/kube-auth-proxy/0.log" Jul 10 20:48:16.790993 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:16.790960 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wshtg_01fdc1af-61b3-4a78-8b96-2235b760e26d/kube-multus-additional-cni-plugins/0.log" Jul 10 20:48:16.811394 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:16.811366 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wshtg_01fdc1af-61b3-4a78-8b96-2235b760e26d/egress-router-binary-copy/0.log" Jul 10 20:48:16.831347 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:16.831316 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wshtg_01fdc1af-61b3-4a78-8b96-2235b760e26d/cni-plugins/0.log" Jul 10 20:48:16.851461 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:16.851431 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wshtg_01fdc1af-61b3-4a78-8b96-2235b760e26d/bond-cni-plugin/0.log" Jul 10 20:48:16.871496 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:16.871468 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wshtg_01fdc1af-61b3-4a78-8b96-2235b760e26d/routeoverride-cni/0.log" Jul 10 20:48:16.892083 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:16.892051 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wshtg_01fdc1af-61b3-4a78-8b96-2235b760e26d/whereabouts-cni-bincopy/0.log" Jul 10 20:48:16.911651 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:16.911621 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wshtg_01fdc1af-61b3-4a78-8b96-2235b760e26d/whereabouts-cni/0.log" Jul 10 20:48:17.007697 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:17.007666 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-s9rr9_e2cf593d-020d-43fa-afd6-d85e2a98c2da/kube-multus/0.log" Jul 10 20:48:17.143085 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:17.143058 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xfmzc_3ac597d8-dc79-46ad-954e-bc34432d454a/network-metrics-daemon/0.log" Jul 10 20:48:17.160941 ip-10-0-143-108 kubenswrapper[2565]: I0710 20:48:17.160903 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xfmzc_3ac597d8-dc79-46ad-954e-bc34432d454a/kube-rbac-proxy/0.log"